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ABSTRACT 

This report focuses on the question of how to 
evaluate workplace education programs. Chapter 1 outlines the various 
purposes that those involved in workplace education programs have 
defined for workplace education and describes program models in place 
to meet those goals. Chapter 2 sets out three sc :arios for 
evaluation at present: informal, internal monito* ,; more formalized 
evaluation by outside evaluators; and more formalized internal 
evaluation by stakeholders. It then describes how the various 
components of evaluation process look within each one. Chapter 3 
describes what a good evaluation looks like and the actions that 
should be taken to create meaningful evaluations. Chapter 4 describes 
workplace educators who have taken a lead in developing the 
collaborative approach and who are revising funding guidelines; 
developing new assessment, curriculum, and evaluation tools; creating 
new opportunities for staff development; and building a constituency 
for workplace education. It also describes others outside the 
workplace education field — literacy educators, social scientists, and 
organizational development and training specialists — who are 
developing tools of possible use to workplace literacy specialists. 
Chapter 5 discusses steps that stakeholders might take to support 
these new developments. (References are listed at the end of each 
chapter. Names and addresses of key organizations and individuals 
cited are appended.) (YLB) 



A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A 

* Reproductions supplied by EDRS are the best that can be made 

* from the original document. 

A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A V A A A A A A A A A A A A A A A A A 



CM 
00 

C4 

M 

So 

o 
uj 



WORKPLACE EDUCATION : 
STAKEHOLDERS' EXPECTATIONS, 
PRACTITIONERS 1 RESPONSES, 

AND 

THE ROLE EVALUATION MIGHT PLAY 



Prepared June 1994 
by 

Paul Jurmo 



This study was funded 
by the 

National Institute for Literacy, 
Washington, D.C. 



U.S. DEPARTMENT OF EDUCATION 
Office of Educai'Onai Research and improvement 
EDUCATIONAL RESOURCES INFORMATION 
,/ CENTER (ERIC) 

h/rhis document has been reproduced as 
(/received from the person or organisation 

* originating it 

D Minor changes have been made to improve 
reproduction quality 

• Points of view or opinions stated m ihis docu 
ment do net necessarily represent oNiciai 
OERI position or policy 



Grant Award #X257A20420 



The views expressed in this document are those of the 
author and not necessarily those of the National Institute 
^ for Literacy. 

vjp Available from Literacy Partnerships, 14 Griffin 

v> Street, East Brunswick, NJ 08816-4806, Tel. 908/254-2237. 

O 

d 2 

ERJC c 

BEST COPY AVAILABLE 



CONTENTS 



Acknowledgements u 

Executive summary. m 

IntroductiQia 1 

History, purposes, and audiences 1 

Research questions and methodology 1 

Chapter 1 : Workplace education: Stakeholders' 

expectations and providers' responses 3 

"Functional context" and other perspectives 3 

What does this mean for those concerned about 

program evaluation? 20 

Chapter 2 : What evaluation currently looks like 

in workplace education programs 28 

What does "evaluation" refer to? 28 

How this chapter is organized 29 

Evaluation, component by component 30 

Chapter 3 : Strengths and needed improvements 

of current evaluation practice 43 

What a good evaluation should look like 43 

What we need to avoid 45 

Chapter 4 : Promising developments 53 

Within the workplace education field , 54 

Elsewhere in the adult literacy field 77 

Within other fields 81 

Chapter 5 : What we can do to develop appropriate 

evaluation practice 92 

Workplace educators 93 

Employers 95 

Unions • 97 

Funders 'and policy makers 98 

Workers 101 

In conclusion • 103 

APPENDIX: Key organizations and individuals 
cited in this report 



ACKNOWLEDGEMENTS 



For their help in the preparation of this document, I thank the 
following: 

The National Institute f or Literacy (NIFD . which provided 
funding for this study and a related handbook and series of 
case studies. 

Laura Sperazi. co-director of this NIFL project and pioneer in 
much of the new thinking represented in this study. 

Sue Folinsbee, another "pioneer" who gave valuable guidance 
from a Canadian perspective. 

Others who provided information and feedback: Ira 
Baumgarten. Mar ilyn Gillespie. Susan Imel. Toanne Tones, 
Glenda Lewe. N anette Brey Magnani. Muriel Medina. Tan 
Newman. Dolores Perin. Sharon Ramirez. David Rosen. Tony 
Sarmiento. T ee Shore. Sue Schurman. Wavne Werbel. and Carol 
Y ou n g . 

Olga. Nicole, and Eric for their patience. 



Paul Jurmo 

Literacy Partnerships 

14 Griffin Street 

East Brunswick, NJ 08816-4806 

908/254-2237 

June 1994 



er|c 



ii 

4 



EXECUTIVE SUMMARY 



WORKPLACE EDUCATION : 
STAKEHOLDERS' EXPECTATIONS, 
PRACTITIONERS' RESPONSES, 
AND 

THE ROLE EVALUATION MIGHT PLAY 

In 1993 and 1994, the National Institute for Literacy funded 
workplace education researchers Laura Sperazi and Paul Jurmo to 
refine a collaborative evaluation methodology for workplace basic 
skills programs. This study, prepared by Paul Jurmo, is one of three 
documents developed under that project. 

Written for key stakeholders in the workplace education field, 
it provides an overview of (1) purposes for and approaches to 
workplace education; (2) how workplace literacy programs are 
evaluated; (3) strengths and needed improvements of current 
evaluation procedures; (4) recent attempts to develop a new, 
"collaborative" approach to workplace education, and (5) steps 
stakeholders can take to strengthen the field. As such, this document 
serves as a background piece to the other two project documents - 
case studies and a guidebook — which focus more directly on the 
"team" or "collaborative" approach to evaluation. 

This study draws on literature from the workplace literacy 
field and related disciplines, and from interviews with workplace 
educators. Key findings are summarized below: 

Workplace education: Stakeholders' expectations and 
providers' responses 

Stakeholders in workplace basic education programs include 
learners, employers, unions, education providers, and funders. They 
have many expectations for workplace education. Some want to 
improve workers' abilities to perform specific job-related t£ sks. 
Some see education as one of many initiatives a workplace can 
undertake to (1) transform itself into a high-performance, 
continuous-improvement organization and (2) enable workers to be 
active participants in that transformation. 

Some stakeholders hope education programs will help workers 
achieve more purely "academic" goals, improve their self-esteem and 

* • * 
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morale, or perform tasks important to them which may or may not 
be job-related. Some stakeholders look beyond individual 
workplaces and workers and see worker education projects as tools 
for developing the workplace education field or the larger economy 
and society. 

Evaluators of workplace education programs need to 
understand what stakeholders hope to accomplish. They can thereby 
better understand who it is they are dealing with, avoid 
unproductive conflicts, and respond to stakeholders' legitimate 
concerns. 

What evaluation currently looks like in workplace 
education programs 

Research and policy papers state that evaluation should be an 
intrinsic part of all workplace education programs, to provide 
reliable information to those most concerned about program impact 
and needed improvements. In reality, however, programs have 
historically given relatively little attention to evaluation, focusing 
instead on providing direct services. They squeeze in informal 
internal monitoring when they can and, if the budget permits, hire 
an outside evaluator. Most recently, sor~ » programs have created 
internal evaluation teams which take responsibility for collecting, 
analyzing, and reporting information. 

Strengths and needed improvements of current evaluation 
practice 

A solid evaluation should respect stakeholder confidentiality; 
be seen as a positive, useful tool for making constructive decisions 
throughout the life of the program; and produce clear, reliable 
information about program impact and needed improvements. 
Evaluation should be guided by a spirit of inquiry, lifelong learning, 
and continuous improvement. Those involved need to understand 
not only evaluation itself but the special nature of workplace 
education. Audiences should use evaluation reports to refine funding 
guidelines, professional practice, and program goals. 

Evaluators need, conversely, to avoid overlooking the 
information which stakeholders really want. Evaluations should not 
be an afterthought or seen as "for someone else." Evaluation should 
not be limited to "a test." Information should not be of dubious 
credibility, unclear, neglected, or unnecessarily negative in tone. 
Valuable findings should be disseminated widely to those who can 

iv 
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learn from them. Workplace evaluations shouldn't oversimplify the 
relationship between literacy and productivity. Nor should they 
ignore the insights of practitioners, learners, supervisors, and others 
with valuable knowledge. And evaluations should not be given low 
priority and low funding. 

Promising developments 

Innovators in the workplace education field are developing 
new ways of evaluating workplace education programs and, with 
them, new forms of workplace education itself. Many of these 
innovations emphasize collaborative decision-making by 
stakeholders. Education programs are thereby more directly tied in 
with new ways of organizing work which emphasize teamwork and 
continuous improvement. 

Within the workplace education field, some funders are now 
encouraging programs to follow collaborative decision-making 
principles. Practitioners are coordinating all program components in 
an integrated cycle characterized by continuous, collaborative 
decision-making. Stakeholders are being encouraged to first analyze 
the organization's overall human resource needs and policies before 
deciding whether to embark on an employee education initiative. 
Practitioners are creating new forms of individual assessment, 
curriculum development, and evaluation which put greater 
responsibility into the hands of program participants and other 
stakeholders. Practitioners are applying collaborative learning 
principles to their own professional development, by establishing 
peer-support systems. 

In the related fields of adult literacy education, social sciences, 
and training and organizational development, innovators have 
developed assessment, evaluation, and program models which mirror 
many of the innovations being developed by workplace literacy 
educators. 

Each of these disciplines might benefit from greater cross- 
fertilization with the others. 

What we can do to develop appropriate evaluation practice 
Key stakeholders in the workplace education field have much 
to learn from good work done to date and much work to do to 
develop the field further. Workplace educators, employers, unions, 
funders and policy makers, workers, and others need to study, 



clarify their values, develop action plans, build constituencies, get 
resources, do research and development, break down old barriers, 
and practice the collaborative learning and problem-solving we want 
workplaces to adopt. 
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INTRODUCTION 



History, purposes, and audiences 

This study was written as part of a larger project funded in 
1993-1994 by the National Institute for Literacy (NIFL). The larger 
project was co-directed by Laura Sperazi and this author. The 
project built on prior work which the co-directors had carried out in 
workplace education programs in Massachusetts and elsewhere. 

The project's goal was to provide a theoretical foundation and 
practical guidance for workplace educators looking for ways to 
improve the evaluation of workplace basic skills programs. In 
particular, the project presents arguments and practical steps for a 
"collaborative" or "team" approach to planning and evaluation. 

This document focuses in particular on the "theory," the 
thinking which underlies not only the collaborative approach but 
other approaches to workplace education planning and evaluation. 
The author believes that anyone hoping to take on the task of 
creating evaluation relevant to this complex field should first have a 
working understanding of the thinking and models which already 
exist. In addition to laying out what already exists, this study in its 
last chapter also provides practical guidance for key stakeholders ~ 
workplace education practitioners and researchers, employers, union 
representatives, and workers ~ interested in "next steps," figuring 
out where to go from here. 

Research questions and methodology 

This study's five chapters are built around five discussion 
questions: 

1. In the field of workplace basic skills education, what are the 
purposes which stakeholders set for programs and what 
approaches have practitioners developed in response? 

2. What does "evaluation" currently consist of in workplace 
education programs? 

3. What are strengths and needed improvements of workplace 
education evaluation to date? 
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4. What efforts are currently underway (in workplace 
education, adult literacy education, social sciences, and 
organizational development) to develop planning and 
evaluation methodologies which workplace educators might 
now learn from? 

5. What actions might stakeholders in the workplace education 
field now take to strengthen how programs are planned and 
evaluated? 

To respond to the five discussion questions, the author 
reviewed the literature on workplace basic skills education which 
has emerged since the late 1980s. He relied not only on "familiar" 
(commonly-cited) documents found in the ERIC database but on less- 
accessible conference presentations, minutes of meetings, and 
program evaluation reports solicited through an informal "grapevine" 
of workplace educators with whom the author has worked in recent 
years. He focused in particular on sources which dealt with the 
purposes of workplace education programs and the evaluation of 
those programs. He also interviewed a dozen practitioners and 
researchers in workplace education. In selecting his sources, the 
author attempted to present a representative sampling of the 
perspectives and models which currently exist. 

For the first four research questions, relevant information was 
gathered from the above sources and compiled under those four 
questions. As this information was compiled, the author identified 
key themes. These themes became the framework for presenting the 
various ways the field has been responding to the first four research 
questions. 

For the final research question, the author reflected on the 
information compiled for the first four questions and on his own 
experience, to map out possible steps which stakeholders might now 
take to improve how workplace education programs are planned and 
evaluated. He circulated a draft of the resulting report to a half- 
dozen reviewers. He then incorporated their feedback into the final 
report. 

The author provides some answers to the five research 
questions. He hopes that readers will, in turn, develop their own 
answers to these questions and add them to the healthy dialogue our 
field now needs. 



Chapter 1 



WORKPLACE EDUCATION : 
STAKEHOLDERS' EXPECTATIONS 
AND 

PROVIDERS' RESPONSES 



Stakeholders in workplace basic education programs include 
learners, employers, unions, education providers, and funders. They 
have many expectations for workplace education. Some want to 
improve workers' abilities to perform specific job-related tasks. 
Some see education as one of many initiatives a workplace can 
undertake to (1) transform itself into a high-performance, 
continuous-improvement organization and (2) enable workers to be 
active participants in that transformation. 

Some stakeholders hope education programs will help workers 
achieve more purely "academic" goals, improve their self-esteem and 
morale, or perform tasks important to them which may or may not 
be job-related. Some stakeholders look beyond individual 
workplaces and workers and see worker education projects as tools 
for developing the workplace education field or the larger economy 
and society. 

Evaluators of workplace education programs need to 
understand what stakeholders hope to accomplish. They can thereby 
better understand who it is they are dealing with, avoid 
unproductive conflicts, and respond to stakeholders' legitimate 
concerns. 

"Functional context" and other perspectives 

In their 1992 review of literature on workplace literacy 
education, Susan Imel and Sandra Kerka identified two purposes for 
employee basic skills programs. Since the mid-1980s, workplace 
education tended to be seen as a means of helping employees 
perform particular "high-priority job tasks as determined by 
management" (Imel and Kerka, 1992, p. 4). More recentiy, however, 
some in the field have conceived of programs as a means of changing 
not just "the behavior of individual employees but of the larger work 
organization as well" (Imel and Kerka, 1992, p. 4). In this new view, 
workplace basic skills programs should "reinforce the critical 
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thinking and teamwork required to transform workplaces into high- 
performance, continuous improvement organizations" (Imel and 
Kerka, 1992, p.8). 

These categories represent two important ways of looking at 
workplace education. The former perspective ~ commonly referred 
to as the "functional context" approach - has guided U.S. federal 
funding for workplace literacy programs; it shaped public thinking in 
the rush of media coverage given to workplace literacy in the late 
1980s; and it became the standard against which many programs 
measured themselves. The latter perspective — which we will call 
the "collaborative" approach ~ is more recent, but it appears to be 
attracting increasing support from those linking workplace education 
to more collaborative ways of organizing work. 

These two categories don't, however, capture other goals which 
stakeholders have set for worker education. These other goals 
include more-traditional "academic" objectives, increased learner 
self-esteem, learner-defined contextualized uses of literacy on and 
off the job, development of the workplace education field itself, and 
economic and social development. 

This report focuses on the question of how to evaluate 
workplace education programs. In any such discussion, it is 
important to first understand what those involved in the program 
hope the program will achieve before we can determine whether it 
in fact has achieved the intended outcomes. 

To clarify the options open to programs, outlined below are the 
various purposes which those involved in workplace education 
programs — researchers, funders, employers, unions, educators, and 
program participants ~ have been defining for workplace education. 
Also described are program models which have been put in place to 
meet those goals. 

Improving individual worker performance of specific job 
tasks 

Project Literacy U.S. - commonly called the "PLUS Campaign" 
was a collaboration of the ABC and PBS television networks which 
began in the mid-1980s and, with varying focal points, has 
continued until this time. Its original purpose was to focus the 
public's attention on the adult literacy issue (although in more 
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recent years it has shifted its focus to youth). In its second year, 
it focused in particular on workplace literacy. The campaign 
fueled the creation of "business breakfasts;" governors' task 
forces; intensified coverage in the print media; state and federal 
workplace literacy initiatives; and special pilot projects of trade 
associations in the banking, printing, textile, and home building 
industries. 

These workplace literacy activities were supported by the 
Business Council for Effective literacy (an organization set up 
specifically to foster corporate involvement in adult literacy 
efforts) and by other business-oriented organizations like the 
American Society for Training and Development, the National 
Alliance of Business, the Society for Human Resources 
Management, the American Newspaper Publishers Association, 
and the American Management Association. 

These initiatives generally emphasized the impact of illiteracy 
on the "bottom line" of companies. To counter American workers' 
perceived skills deficits, companies were urged to invest in 
workplace education programs which helped workers strengthen 
the basic communications skills they needed to perform their 
current jobs and possibly future jobs. 

This focus on specific job skills was based on research done in 
workplace literacy programs - much of it carried out in the 
military - where improved job performance was the stated goal. 
Proponents of this job-specific approach ~ which came to be 
referred to as the "functional context" approach ~ argued that it is 
simply more efficient to teach people the skills they need to use 
on a day-to-day basis and then let them practice those skills in 
their day-to-day work (Sticht, 1987). 

They contrasted job-specific instruction with what they termed 
"general" literacy instruction, the teaching of abstract, de- 
contextualized skills traditionally done in formal schools. 
Functional context advocates said that "general literacy" programs 
are inappropriate for workplace settings because they were based 
on a logic developed for children in school: teach abstract "skills" 
divorced from real, meaningful application with the hope that ~ 
somehow, someday ~ the learners will be able to transfer those 
skills to real applications (Secretary's Commission on Achieving 
Necessary Skills, 1991). 
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The arguments presented by advocates for a job-specific focus 
are summarized below (Mikulecky and Lloyd, 1993): 



1. " Research ": Research shows learners learn best through 
regular practice of real applications to serve meaningful 
purposes in real contexts. "General literacy" is either not 
grounded in any context or focuses on contexts which are not 
relevant in a workplace. 

L 

2. " Practicality ": Given the miniscule instructional time 
available in most workplace programs, learners must focus on 
high-priority, attainable learning objectives, which they can 
reinforce through practice in their daily work lives. "General 
literacy" is too unfocused, trying to cover too many objectives. 

3. " Politics ": To convince employers to invest in education, we 
must show that their bottom line economic interests are 
served. "General literacy" programs don't sufficiently improve 
job performance to justify investments in them. 

The new attention to workplace literacy created a demand for 
literacy practitioners able to operate workplace programs. Few 
had set up workplace literacy programs before. Those who had 
tended to follow more traditional, academic models which used 
the general literacy approach described above. 

The arguments for a functional context approach seemed 
reasonable, and newcomers wanted practical guidance on "how to 
do it." They turned to the pioneers of the functional context 
approach for help. Several manuals showing how to set up 
workplace literacy programs received wide circulation (Business 
Council for Effective Literacy, 1987; Carnevale, Gainer, and 
Meltzer, 1990b; Philippi, 1991; U.S. Department of Labor and U.S. 
Department of Education, 1988). The functional context approach 
was further disseminated in popular workshops all over the 
country. 

Practitioners hungry for practical, "how to" guidance gave 
special attention to the question of how to do a literacy task 
analysis (also known as "a literacy audit"). Although the details of 
how to do an "LTA" varied from practitioner to practitioner and 
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site to site, the LTA process tended to have these characteristics 
(Philippi, 1988, 1991): 



Decisions made primarily by education specialists and 
managers : It is assumed that programs will focus on job tasks 
identified by employers as being critical to job performance. 
Education specialists then do most of the work of investigating 
those jobs and workers' abilities to perform them and then 
developing a curriculum to fill in the gaps between job 
demands and worker skills 

Most decisions made during a single period prior to 
implementation of instruction : Needs analysis and curriculum 
design work are done primarily before the program gets under 
way. 

Emphasis on breaking down jobs into tasks and skills: The LTA 
process breaks jobs into tasks, which are then sequenced and 
prioritized by difficulty and frequency of use. Tasks are then 
further broken down into steps, substeps, and single behaviors. 
For each task, the required cognitive reading process and 
concepts required for successful performance are identified. 
These become focal points -- learning objectives -- for 
instruction. Instructional activities are further categorized by 
ability level, time required, and other parameters. 

Emphasis on helping learners locate and use information 
developed by others: The LTA focuses heavily on reading tasks 
in which the worker locates or uses information imbedded in 
job documents developed by others. 

In the mid-1980s, the U.S. Department of Education adopted 
the functional context approach, promoting it in the Department's 
The Bottom Line workplace literacy handbook. The legislation 
which created the Department's National Workplace Literacy 
Program stated the broad goal that funded projects should 
"improve the productivity of the workforce through improvement 
of literacy skills needed in the workplace." The Department 
interpreted this to mean that funded projects follow the functional 
context job-specific methodology. As the largest single source of 
funds for the field, the Department thereby made "functional 
context" the criterion against which the field measured itself 
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(Sarmiento, 1993; Schultz, 1992; U.S. Department of Education, 
May 1992). 



Helping organizations shift toward high-performance 
teams 

Emerging in the early 1990s came a different interpretation of 
"the workplace context," "employee basic skills," and how to create 
employee basic skills programs relevant to the emerging 
workplace context. This view held that a new workplace was 
evolving in which workers need to be able to perform a wide 
variety of jobs, to be able to make decisions, to communicate with 
many different people for various purposes, to take initiative and 
solve problems, and to work in teams. This kind of role was seen 
as significantly different from what was expected of workers in 
more-traditional, top-down, assembly-line types of jobs. In the 
"old" workplace, workers were expected to repetitively do the 
same tasks over and over, not ask questions, follow orders given 
by others further up the decision-making hierarchy, and not take 
too much initiative. 

"Basic skills" in the old workplace often was seen as reading 
information and directions developed by others and solving 
problems as interpreted by others in the context of relatively 
static jobs. In the new workplace, "basic skills" essentially meant 
the ability to create new knowledge — a new understanding of 
workplace needs ~ and then use that knowledge to continually 
improve how work is done. 

According to this emerging view, workplace education 
programs should help workers develop those new kinds of skills 
and a mindset which values continuous self-improvement, 
improvement of the organization, and active, collaborative 
problem-solving. Education programs should themselves be 
organized along the lines of the collaborative decision-making 
processes used in the new workplace (Jurmo, 1994; Sarmiento, 
1991; Pritz and Imel, 1993; Schultz, 1992; Stein, 1993). 

like the "functional context" proponents who preceded them, 
advocates of this alternative approach were, on one hand, reacting 
against an existing approach (that is, the functional context 
approach) while building on their own experience trying to 
develop practice meaningful in workplace contexts. In this case, 
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I the workplace contexts they were most interested in working in 

differed from the more traditional workplaces in which functional 
context programs evolved. 

, Those advocating an instructional approach tied more directly 

[ into transforming workplace cultures came from several sources: 

I Educators who had tried the functional context approach and 

' found it too limiting : One of the positive outcomes of the 

workplace literacy pilot projects funded by the federal 
I government and other sources was experience which showed 

what approaches seemed to work in particular contexts ~ and 
i which didn't. Some practitioners who had tried the functional 

| context approach emerged from that experience saying they 

felt constrained trying to teach particular skills which - 
| especially in the rapidly changing workplace - were soon 

I irrelevant if they were ever relevant at all. Many of these 

practitioners also came from whole language, community 
| based, and other literacy education backgrounds which 

emphasized active learner involvement in curriculum design 
. (Gowen, 1992; Jurmo, 1991; Pritz and Imel, 1993). 

Schultz, for example, states that the functional context 
| approach and its heavy reliance on LTAs, have 

. . . become axiomatic with workplace education 
I programs. What is striking about literacy audits is that 

1 they almost always lead to lists of skills and subskills 

rather than to a broader understanding of teaching and 
J learning literate practices. Furthermore, and perhaps 

even more importantly, the dominant ideology which 
| supports this single way of conceptualizing the 

I curriculum necessarily limits the possibilities for teaching 

and learning in these programs . . . These ways of 
I describing teaching and learning, and of describing 

' teachers and learners reinforce an essentially Tayloristic 

view of work, a perspective which advocates a system of 
I mass production based on breaking work into its smallest 

and most basic elements and defining jobs narrowly so 
I that they are relatively easy to learn. Thus, while U.S. 

| companies are introducing new forms of work 

organization, with new participation structures and 
| patterns of interaction, many are choosing to teach and 

I . 9 17 

ERIC 



evaluate their employees with traditional methods. 
Reorganized companies are asking workers to become 
active learners in the workplace and passive students in 
the classroom. The reorganization of work and 
workplaces calls for new conceptions of teaching and 
learning as active, constructive process rather than the 
transmission of skills (Schultz, 1992, pp. iv-v). 

A report from a 1992 close-out conference of NWLP 
project directors contains a similar quotation from one director: 

The rigid view of job-specific curriculum development 
held by many leaders in workplace literacy is dangerous. 
Job-specific curriculum development makes sense 
because it utilizes the concept of integrating prior 
knowledge into instructional design. The danger, 
however, lies in the belief that it is the only way to 
develop good curriculum. This rigid view is based on a 
small amount of research conducted by a few people who 
have done adequate research, but with a very, very 
narrow focus. The need for funding more research in 
what works in workplace literacy is crucial (Evaluation 
Research, 1992, Appendix A). 

Those calling for greater quality of work life for workers : The 
shift toward participatory decision-making in workplaces 
didn't, as many might assume, begin with the Total Quality 
Management (TQM) movement which emerged in the 1980s. 
Employee involvement was a foundation stone of the "quality 
of work life" (Q.WL) movement of the 1970s which first began 
reshaping how people thought work should be done. The QW1 
movement put a special emphasis on making work more 
rewarding for workers, in terms of reimbursement, benefits, 
equal opportunity, as well as general job satisfaction. 

Although QWL as a concept became overshadowed in the 
1980s as the TQM movement presented employee involvement 
as a means of "improving the bottom line," it nonetheless has 
remained a motivation for many now involved in trying to shift 
workplaces toward a "high performance" model. That includes 
many of those calling for a more-participatory approach to 
worker education. They argue that education which prepares 
workers for the more-active decision-making roles expected of 
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them in the new workplace not only can make for a technically 
more efficient workplace but one which ensures for them a 
greater quality of life on and off the job (Sarmiento and Kay, 
1990; Sarmiento and Schurman, 1992; Schurman, 1993). 

As Stein (May 1993) points out, however, worker 
involvement and a continous improvement ethic don't 
automatically guarantee workers job security. Efforts to 
transform workplaces along high performance lines must 
include a commitment to equity and justice for workers, as 
well. 

Those who see education as a wav of producing "hig h 
performance." productive, and competitive workplaces : As the 
1980s came to a close, there was an explosion of books and 
articles arguing that, if U.S. business was to regain its 
competitive edge and head off economic and social disaster, it 
must re-organize the way work is done. Businesses should 
transform themselves and become "high-performance work 
organizations" guided by a "continuous improvement" ethic and 
total quality management principles. This new way of viewing 
work was to place high value on active worker participation in 
collaborative decision-making, flattened hierarchies, greater 
labor-management cooperation, commitment to ongoing 
learning, and other innovations (Tomasko, 1993). Education 
and training were seen as vital elements of such organizations 
(Aubrey and Felkins, 1988; Berger, Dertouzos, Lester, Solow, 
and Thurow, 1989; Carnevale, Gainer, & Meltzer, 1990a; 
Commission on the Skills of the American Workforce, 1990; 
U.S. Congress, 1990). 

Senge argues that, if companies truly want to transform 
themselves in a high performance direction, they must become 
"learning organizations." In such a workplace, "personal 
learning and organizational learning" are connected, there are 
"reciprocal commitments between individual and organization" 
and there is a "special spirit of an enterprise made up of 
learners" (p.8). Senge argues that "team learning is vital 
because teams, not individuals, are the fundamental learning 
unit in modern organizations" (p.8). He says: "The discipline of 
team learning starts with 'dialogue,' the capacity of members of 
a team to suspend assumptions and enter into a genuine 
'thinking together'" (p.8). "A learning organization is a place 
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where people are continually discovering how they create their 
reality. And how they can change it" (p. 12). 

He contrasts tliis "decentralized, nonhierarchical 
organization dedicated to the well-being and growth of 
employees as well as to success" (p. 15) with "traditional 
authoritarian 'controlling organizations'" (p.5). Under the 
traditional system, "we are taught to break apart problems, to 
fragment the world. This apparently makes complex tasks and 
subjects more manageable, but we pay a hidden, enormous 
price. We can no longer see the consequences of our actions; 
we lose our intrinsic sense of connection to a larger whole" 
(p.l). The result is "our inability to grasp and manage the 
increasingly complex systems of our world" (p. 14). 

A few sources have focused more specifically on where 
basic skills education might fit into this new way of organizing 
work. Sarmiento (1991, p. 2) states: 

Planning a workplace literacy program in a "high skill," 
high performance work organization is unlikely to be a 
top-down, prescriptive process. Instead, basic skills 
problems are recognized and handled through the same 
participatory process and structure that exists in the 
workplace to identify and solve other kinds of problems 
related to production or labor-management issues. 

He contrasts such a program to one operating in a 
traditional, top-down, "Tayloristic" workplace where 

. . . workplace literacy programs can be designed to 
develop a dispensable, disposable workforce to meet the 
low literacy and job skill demands of an employer 
pursuing scientific management principles and the "low 
wage" path (Sarmiento, 1991, p. 3). 

Similarly, Stein and Sperazi (1991, pp. 1-3) contrast 
worker education in traditional and high performance 
workplaces. In traditional workplaces, the 

. . . company does not have a long term strategy that 
integrates education and training into an overall business 
plan . . . goals for education and training are short-term, 



problem-centered . . . (there is a) presumed conflict 
between education and production . . . top management is 
not invested in the program's goals . . . neither workers, 
supervisors, or unions are involved in planning, 
implementing, and evaluating the program . . . decisions 
on what and whom to teach are based on an audit of job 
task specific deficits . . . testing is separate from 
instruction . . . content of instruction tends to be 
narrowly job specific . . . and there are no plans for 
institutionalization of program. 

Stein argues that education and learning be seen as 
intrinsic parts of any high performance organization, rather 
than as something tacked on to get a company through a crisis. 
She describes one workplace education program which has 
evolved from a focus on life skills to a focus on job-specfic 
content and, finally, to a focus on continuous improvement 
tools. The ESL instructor is now training workers to serve as 
peer-teachers of fellow workers out on the shop floor, thereby 
reinforcing the notion that learning occurs every day in every 
aspect of work, not just in the classroom. She argues that we 
should be looking at the natural ways people learn on the job 
- typically in colaboration with more-experienced co-workers 
~ and help workers use those existing, everyday, natural 
learning processes rather than trying to fit workers into a 
traditional classroom model. In this alternative model, we 
would 

. . . expand the walls of our classroom, to see the 
workplace itself as the classroom and our role as teachers 
to facilitate the learning that is going on thoughout it, to 
make what we do part of the action, part of the process of 
continuous improvement (Stein, May 1993, p. 7). 

Hull (1991) cites ethnographic research on how people 
naturally help each other with literacy tasks in workplace and 
other environments. She suggests an "apprenticeship" model of 
learning in which 

. . . workers who need to carry out a task involving 
complex literacy skills learn on the job with someone who 
can already perform that task . . . rather than assuming 
that structures and practices for learning literacy must be 



21 

13 



imported from school-based models of teaching and 
learning, we might do well to study workplaces and 
communities to see what kinds of indigenous structures 
and practices might be supported and built upon (pp. 28- 
29). 

A growing number of workplace educators are currently 
moving beyond the level of theory and developing programs 
which reflect the above collaborative learning principles. Their 
efforts are described under "Promising Developments" in 
Chapter 4 of this report. These practitioners cite the following 
kinds of principles: 

• Workplace education efforts should promote the 
development of workplaces which are both efficient and 
characterized by a high quality of work life for all 
employees. 

• Learners focus on understanding and shaping the larger 
organization and less on mastering discrete tasks and 
information developed by others. 

• Learners develop skills and knowledge they can apply not 
only to the workplace but to other contexts, as well. This is 
done for two reasons. It encourages learners to strengthen 
their skills through continuous practice in many meaningful 
daily contexts. It also prepares workers to participate 
actively both within and outside the workplace, thereby 
contributing to the creation of a stronger and more 
democratic economy and society. 

• Instructional content and format are flexible, continually 
revised in response to needs and opportunities identified by 
stakeholders. 

• All stakeholders ~ especially learners - have 
responsibility for deciding content and format of program 
activities. Special procedures are established to enable all 
stakeholders to participate actively in decision-making. 

• Improving employee basic skills is seen as one component 
of a long-term strategy for changing the larger 
organizational culture. 
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• All program elements are linked together (not piecemeal) 
and managed via a collaborative decision-making process 
involving all stakeholders. 

• Learners are encouraged to use the skills they are 
developing, and stakeholders make changes in the 
workplace necessary to enable learners to do so. 

• Emphasis is on helping learners create knowledge and 
communicate it rather than merely using information 
developed by others. 

• All stakeholders are invested in developing the potential 
of the entire workforce (not just managers). 

• Education is seen as a means — not an obstacle ~ to 
improved job performance. 

• Individual assessment is integrated into instruction and 
results are used in a confidential, non-threatening way. 

• Outcomes are measured in terms of changes not only in 
individual workers but in the organization as a whole. 

• The dignity and diversity of all stakeholders is respected 
(no "bashing" of anyone). 

• Stakeholders avoid focusing on "deficits." 

• Education efforts borrow from those involved in work re- 
organization, adult education, and other fields. 

Helping workers to achieve academic goals 

Some stakeholders — including employers, union 
representatives, potential and actual participants, and educators 
~ take what might be considered a fairly traditional, "academic" 
view. They think that workers should concentrate on developing 
their basic abilities in the 3Rs and perhaps earn their GEDs. With 
such a foundation of basic academic skills, participants will then 
presumably be able to better handle the tasks they face on and 
off the job. The emphasis in this case is not so much on 
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immediate improvements in job performance as on providing 
workers with an educational benefit which they might use to 
improve their job performance or their lives in general. Such a 
perspective is often made by those who think of "literacy" as the 
3Rs learned in school or who are unaware of - or don't buy into 
— the arguments for a more-contextualized approach to adult 
literacy instruction. 

This perspective on what programs should achieve is 
essentially the "general literacy" approach rejected by "functional 
context" proponents. In an informal survey of workplace literacy 
programs taken in the mid-1980s, the Business Council for 
Effective Literacy found that most programs at that time took this 
approach. As stated under "functional context" above, U.S. 
programs have since that time been required by some funders to 
adopt a functional context approach, despite the fact that many 
stakeholders -- especially learners -- ask for the workbooks, 
vocabulary drills, and other "academic type" activities they are 
familiar with from their school days (Gowen, 1992; Jurmo, 
November 23,1993; Medina, 1993; Soifer, Irwin, Crumrine, 
Honzaki, Simmons, & Young, 1990; Young, 1993). 

In Canada, workplace educates have felt freer to use a wider 
range of instructional activities - including GED preparation and 
other curricula not directly job-related — possibly because the 
"functional context" approach has not been so heavily promoted as 
in the U.S. (Garlick-Griffm, 1993). 

Improving worker self-esteem and morale 

Some stakeholders ~ including many employers ~ think 
programs should focus on building learner "self-esteem," their 
sense of self-worth. They argue that low-literate workers deserve 
a second chance at attaining the sense of self-worth which comes 
from having a sound education (Draper, 1991). It is often 
assumed that, in addition to helping the workers grow as 
individuals, improved self-worth might also have a positive 
impact on learners' job-performance. For example, learners might 
be more willing to participate in training activities, communicate 
' with fellow workers, and try for promotions. A 1992 report from 
the U.S. Department of Education captured this perspective in a 
quotation from a workplace educator: 
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A year ago Tom had little confidence in himself and no pride in 
his appearance or work. As a result of the success he 
experienced in training, he's become highly productive and 
takes pride in his work. His entire appearance and demeanor 
have changed, too. With his new-found confidence and self- 
esteem, he takes advantage of every in-house training 
opportunity placed in front of him (U.S. Department of 
Education, 1992, p.8). 

"Improved self-esteem" has sometimes been denigrated as too 
soft a benefit to measure and not tangible enough to warrant 
private or public funding. This is despite the fact that employers 
often cite this as a goal when asked what they hope an education 
program might accomplish (Garlick-Griffin, 1993; Jurmo, 
November 20,1993; Jurmo, November 23, 1993; Program Planning 
Consultant Group, 1992). 

Some research supports increased self-esteem as a goal in and 
of itself, however. In this case, self-esteem is defined more in 
terms of a sense of "self -efficacy," a belief that one can succeed 
and that it's worth making the effort to try something new. 
Without such a sense of self-efficacy, this research suggests, a 
worker is unlikely to either benefit from an education program or 
be willing to participate in any other activity aimed at improving 
the workplace and his place in it. 

Adults with a negative impression of their own abilities are not 
likely to attempt literacy away from the supportive 
environment of a classroom and nurturing instructor . . . 
(Learners) need to see themselves as capable of attempting 
more literacy and practicing more on their own, as opposed to 
avoiding literacy tasks and literacy practice (Mikulecky and 
Lloyd, 1993, p. 83). 

Many programs recognize this and intentionally try to build 
learner self-esteem (Soifer, Young, & Irwin, 1989). One program 
points out, however, that reinforcing learner self-confidence must 
be built into every program activity, by showing respect, listening 
to what learners have to say, and building on their knowledge. 
This perspective on self-esteem was central to the program 
developed by Eastern Michigan University, the United Auto 
Workers, and Ford Motor Company at the Ford plant in Ypsilanti, 
Michigan: 
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The workers develop self-confidence when they see that then- 
own knowledge and interests are valuable to themselves and to 
others. Self-confidence is not "taught" in isolated, artificial 
"self-esteem" exercises, but developed in the context of real 
uses of meaningful written and spoken language in familiar 
social and working contexts (Business Council for Effective 
Literacy, July 1988, p. 10). 

Enabling workers to perform worker-defined tasks 

Some stakeholders agree that workplace programs should help 
participants deal with practical uses of literacy but not necessarily 
uses which come from workers' immediate jobs. Rather, the 
choice of deciding what functional uses to focus on is primarily in 
the hands of the learner, and the contexts might be the workplace 
or other situations where the learner uses communications skills. 
In many cases, the context chosen is the participant's family, 
where the participant hopes to be able to understand messages 
sent home by the child's school, talk with teachers, and be on a 
par with ~ and/or maintain a position of authority vis-a-vis ~ 
her/his child or spouse. In some cases, participants want to 
improve their chances for a better job- either in their current 
workplace or elsewhere. 

Those saying that learners should be the ones who define the 
practical uses of literacy to focus on make the following 
arguments: 

• Most workers already know their jobs fairly well. They are 
also tired of dealing with job-related matters every day. In 
some cases they don't really need to do much reading or 
writing in their current jobs, and there is little or no likelihood 
that they could ever move into a job - at least within then- 
current workplace - where more reading and writing would 
be needed. They are for these reasons not likely to be 
motivated by a curriculum which focuses only on their current 
jobs (Business Council for Effective Literacy, July 1989; Gowen, 
1992; Young, 1993). 

• To capture learners' motivation, they must see the program 
as serving their actual interests. These might be job-related, 
but they might also be other concerns not directly related to 
their jobs, including family matters, using the local 
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transportation and health care systems, hobbies, and 
preparation for better jobs perhaps outside the company (Turk 
and Unda, 1991). 

• If a program can respond to learner interests and help them 
see education as a meaningful tool for serving their own needs, 
learners are more likely to be interested in pursuing further 
education. Instilling in employees such an interest in learning 
is vital for any company interested in becoming a "learning" 
organization. 

Contrary to the assumption that all employers think workplace 
education programs should focus only on job-related uses of 
literacy, employers frequently state that they would like their 
employees to be able to use literacy in any way they need it, 
whether job-related or not. Employers frequentiy say they hope 
participants will be able to help their children with schoolwork. If 
participants also are able to use their new literacy skills on the 
job, so much the better. In such cases, literacy education is seen 
more as a personal benefit to workers, with secondary objectives 
of enhancing worker morale and job performance (Jurmo, 
November 20,1993; Jurmo, November 23,1993). 

Contributing to the development of the workplace 
education field 
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Funders ~ including public funders at the federal and state 
levels as well as a few private sources like trade associations 
which have sponsored development of resources for particular 
industries ~ tend to see their funding as an investment in the 
longer-term development of workplace education resources 
(Kutner et al, 1991; U.S. Department of Education, May 1992). 

Some individuals involved ~ particularly adult educators ~ 
see their work as contributing to their own professional growth 
and to the development of practice which can also be used, by 
others (Evaluat 5 ->n Research, November 1992; Jurmo, November 
20,1993; Jurmo, November 23,1993). 
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Economic and social development of workers and their 
communities 



Supporters of workplace education programs sometimes justify 
putting public money into private companies by saying that 
worker education programs can contribute to the economic and 
social development of workers and their communities. Programs 
are conceived as a means of giving a "second chance" to people 
who lacked opportunities for quality education as children. This 
not only helps individual workers to retain current jobs and 
qualify for better ones; it also strengthens the economy by shoring 
up local companies and stabilizing the taxbase and general quality 
of life in communities (Chisman, January 1989; Commission on the 
Skills of the American Workforce, 1990). 

Some who focus on community development hope that 
workplace education will enable workers to participate more 
actively in their families and in other community institutions, 
thereby stemming the erosion of those institutions (Jurmo, 
November 23, 1993; Turk and Unda, 1991). 

Some concerned with issues of diversity and equity in the 
workplace and beyond would like workplace education programs 
to enable workers to respond constructively to increasingly 
diverse co-workers and clienteles. They hope that, by so doing, 
workers will also be able to live productively with diverse groups 
out in the community (Rayman and Sperazi, 1990; Shore, 1993; 
Sperazi and Astrein, 1988). 



What does this mean for those concerned about 
program evaluation? 

The fact that stakeholders have such a range of goals for 
workplace education programs is significant for those concerned 
about evaluation of those programs. The following are reasons why 
we should be concerned: 
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True "partnerships" must respect all stakeholders' goals . 

Funders typically portray workplace education programs as 
"partnerships" between a number of stakeholder groups. A 
stakeholder should be seen as someone who is investing 
something (time, money, facilities, reputation, etc.) in the program 
and hopes for something in return. 

Programs often falter when planners don't listen to what 
stakeholders want and as a result don't develop programs which, 
respond to stakeholders' interests. "Responding" doesn't 
necessarily mean trying to fill every need even if the needs can't 
reasonably be met given existing resources. Rather, "responding" 
might mean honestly telling a stakeholder "we can't do what 
you're asking for with the time and resources we have." Such a 
negotiation process produces trust, an understanding of what the 
program can achieve, and a sense of ownership among 
stakeholders. They are thus more likely to support the program. 

A clear negotiation of stakeholders' expectations is also 
important for those responsible for program evaluation, because 
an agreed-upon set of goals provides criteria to judge the program 
by. Also, if stakeholders feel the evaluation is focusing on 
determining whether their expectations are being met, they will 
be that much more likely to pay attention to ~ serve as an 
audience for — the evaluation's findings. 

Stereotypes can strangle creativity . 

There is often a tendency to pigeonhole the above-described 
perspectives and view options in "either/or" and stereotypical 
terms. Some assume, for example, that a workplace education 
program can achieve only one thing, when in fact it might be 
structured to achieve a number of objectives at once. 

Some also assume that everyone in a particular stakeholder 
group thinks the same way and has the same goals for a worker 
education program. That is not at all true. "Employers," for 
example, might include production managers who have a "bottom- 
line" orientation and want an education program to improve 
workers' performance of particular job tasks. But "employers" 
might also include human resource managers who are more 
concerned about employee morale and see an education program 
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as a way of strengthening worker self-confidence and personal 
growth. 

Programs need to have means for all stakeholders to air their 
expectations and come to some kind of agreement on a reasonable, 
achievable set of common goals. 

Goal-clarification can help us understand the values and 
assumptions which stakeholders bring to a program 

When stakeholders state what they expect a workplace 
education program to achieve, they reveal something about how 
much experience they've had with the workplace education issue. 
They also demonstrate how they view workers, how workplaces 
should operate, and the role of education in that organization's 
culture (Draper, 1991; Schultz, 1992; Waugh, 1991). 

Those of us concerned with designing an appropriate 
evaluation strategy for a program might, by asking stakeholders 
to state their expectations, find that a stakeholder doesn't know 
much about workplace education; is looking for a narrow, quick 
fix; thinks of employees as having deficits (even if it's not clear 
what those deficits are); and is operating on assumptions which 
we are not comfortable with. Or we might find that we are 
essentially in agreement with the stakeholders and that they are 
open to ~ and have already adopted ~ many of the values and 
assumptions we bring to the program. 

Negotiating stakeholders' expectations helps bring those values 
and assumptions out in the open where they can be discussed 
before we make a commitment to working with the parties 
involved. This can help us avoid potential conflict further down 
the line in our working relationship with the stakeholders. Such 
negotiation can also help us clarify our own personal vision and 
the collective vision we will need to work effectively together. 

. . . the discipline of continually clarifying and deepening our 
personal vision, of focusing our energies, of developing 
patience, and of seeing reality objectively ... is an essential 
cornerstone of the learning organization — the learning 
organization's spiritual foundation. An organization's 
commitment to and capacity for learning can be no greater 
than that of its members (Senge, 1990, p. 7). 
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Chapter 2 



WHAT EVALUATION CURRENTLY LOOKS LIKE IN 
WORKPLACE EDUCATION PROGRAMS 



Research and policy papers state that evaluation should be an 
intrinsic part of all workplace education programs, to provide 
reliable information to those most concerned about program impact 
and needed improvements. In reality, however, programs have 
historically given relatively little attention to evaluation, focusing 
instead on providing direct services. They squeeze in informal 
internal monitoring when they can and, if the budget permits, hire 
an outside evaluator. Most recently, some programs have created 
internal evaluation teams which take responsibility for collecting, 
analyzing, and reporting information. 



What does "evaluation" refer to? 

How is the term "evaluation" used ? 

In the field of workplace basic skills education, the term 
"evaluation" means different things to different people. The 
relatively few texts on the subject (Alamprese, 1993; Astrein, 
Sperazi, and Steinberg, 1987; Barker, 1991; Evaluation Research, 
1992; Jurmo, May 1993; Mikulecky and Lloyd, 1993; Philippi, 1992; 
Rayman and Sperazi, 1990; Sperazi and Astrein, 1988; Sperazi, 
Jurmo, and Rosen, 1991; Stein and Sperazi, 1991; Sticht, 1991; U.S. 
Department of Education, May 1992) state that evaluation should be 
an intrinsic part of the program planning/development process and 
provide reliable information both to determine program impact and 
guide needed improvements. However, the term "evaluation" is 
more commonly used in more limited ways. 

Funding guidelines and others advocating the functional 
context perspective described in Chapter I of this report often 
describe "evaluation" as a way of generating quantifiable evidence of 
program impact on individual workers' job performance (Sarmiento, 
1993; Schultz, 1992). Many practitioners respond to those funding 
guidelines by thinking they must therefore produce production 
figures, accident rates, or other measures of learner job performance. 



9 

ERIC 



36 

28 



Many practitioners also hang on to the measures of learner gain they 
are familiar with from non-workplace settings: standardized tests. 
They assume that such tests provide some kind of evidence of 
program impact. 

Our definition 

This paper sees evaluation in the following broad terms: 

In workplace basic skills education, evaluation should be a 
process of providing information to stakeholders to enable them 
to (1) clarify goals, (2) track outcomes, (3) identify factors 
contributing to or impeding success, (4) decide whether to 
continue investing resources and, if so, (5) determine what actions 
are needed to meet stakeholders' goals. 



How this chapter is organized 

This chapter is intended to map out what evaluation looks like 
in workplace basic skills programs in North America. We see three 
scenarios for evaluation at present; 

• Informal, internal monitoring: Most programs carry out a good 
deal of de facto, informal evaluation work on a day-to-day basis. 
Practitioners, learners, and other stakeholders monitor learner 
progress and broader program impact through observation of 
learners inside and outside the classroom, informal discussions 
among stakeholders, and review of written and oral classwork. In 
these cases, the responsibility for evaluation is primarily in the 
hands of the provider, and the monitoring is done to help keep 
stakeholders invested in the program and to clarify how the 
program might be adjusted to produce results meaningful to 
stakeholders. 

• More-formalized eval uation bv outside evaluarors: More-formal 
evaluation ~ with more systematic gathering, analyzing, and 
reporting of data ~ has historically tended to occur only where 
program funders require it. In those cases, funders also often 
require that the evaluation be carried out by an outside evaluator 
who, it is assumed, has the technical know-how and objectivity to 
determine whether the program is meeting its stated goals (U.S. 
Department of Education, May 1992). 



• More-formalized internal evaluat ion bv stakeholders: This 
approach to evaluation is a relatively new one. It attempts to 
combine the best of the above two scenarios by keeping the 
control for the evaluation process primarily in the hands of 
stakeholders closest to the program while ensuring technical 
efficiency and objectivity. This kind of evaluation involves 
stakeholders in a systematic process of identifying program goals, 
clarifying what information they want about the program, 
gathering and analyzing that information, and then making 
decisions with it. This approach aims at providing useful 
information while building stakeholder ownership and support for 
the program. 

To further clarify what evaluation currently looks like in 
workplace education programs, below we describe how the various 
components of evaluation process look within these three scenarios. 
The components described are: 

• Institutional framework 

• Stakeholders responsible for evaluation 

• Audiences and purposes 

• The information gathered 

• Resources allocated 

• How information is gathered 

• How information is organized and analyzed 

• How findings are reported 

• How reports are used 

This overview draws on a number of sources, including 
interviews and focus groups with workplace educators, review of 
evaluation reports, and review of recent articles on workplace 
literacy evaluation. 

Evaluation, component by component 

Institutional framework 

While the best-known workplace basic skills programs in the 
United States are those funded by the U.S. Department of Education 
through its National Workplace Literacy Program, numerous other 
programs are carried out independent of that funding source. These 
include efforts funded by state workplace education initiatives (e.g., 
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in Massachusetts and New York), labor unions, and by employers 
themselves. The U.S. Department of Labor has also funded pilot 
programs, and a number of federal agencies (including the U.S. 
military) have funded programs for their own employees. 

Within these institutional frameworks are a range of 
stakeholders ~ people who have invested time, money, or other 
resources in the workplace education program and who expect 
something in return. Primary stakeholder groups include learners, 
employers (both higher-level and supervisory-level managers), 
union representatives, education providers, and funders. 

Stakeholders responsible for evaluation 

Informal, mternal monitoring of programs is done by the 
educators, learners, and other stakeholders most involved in the 
program on a day-to-day basis. Most of the responsibility for 
evaluation is in the hands of the education provider, who is seen as 
having the most expertise and time to give to the program. The 
resulting monitoring is done to help keep stakeholders invested in 
the program and to clarify how the program might be adjusted to 
produce results meaningful to stakeholders. 

More-formal evaluation done by outside evaluators is often 
carried out at the request of outside funders and gives responsibility 
to an outside evaluation specialist. It is assumed that this outside 
< * ^tor has the technical know-how and objectivity to determine 
r the program is meeting its stated goals. 

Team" evaluation places responsibility for the evaluation 
process in the hands of a representative cross-section of program 
stakeholders: higher- and superviscry-level managers, union 
representatives, participating workers, and education staff. The team 
shares responsibility for the evaluation process with a facilitator, 
who may be a team member or from an outside institution. 

Audiences and purposes for evaluation 

Ideally, an evaluation should provide various audiences who 
have an interest in the workplace education program with 
information they can use to make some sort of decision. Typically, 
stakeholders want to be able to make an informed decision about ( 1) 
whether to continue investing in the program and, if so, (2) what 



actions to take to see that goals are met (Barker, 1991; Jurmo, 
Mayl993). 

In reality, those involved in evaluations are not always sure 
how the information they collect will actually be used. Sometimes 
data are collected primarily to fulfill flinders' reporting requirements 
and, once reports are submitted, it is not clear what is done with the 
information (Jurmo, November 23, 1993). Some might fear that 
evaluations are merely a smokescreen, a ritual which someone is 
putting the program through as a way of forcing changes on the 
program (Jurmo, November 20, 1993). 

Outlined below are potential audiences for a workplace 
education evaluation, along with purposes which such evaluations 
might serve. 

Less formal, day-to-day internal monitoring of program 
progress and needed improvements is done primarily for instructors 
and learners. The resulting information showing positive impact 
maintains learner and teacher motivation and demonstrates to 
instructors what activities produce the best results. Conversely, 
information which indicates lack of positive impact and problems in 
the program shows instructors and learners areas which need 
improvement. Such internal monitoring, then, serves three inter- 
related purposes: to maintain stakeholder interest and enthusiasm, 
further clarify goals, and identify steps which instructors, learners, 
and others might take to achieve desired objectives. 

More-formal evaluations done by insiders or outside evaluators 
are typically aimed at audiences external to the instructors and 
learners normally seen as the "core" of the program. The external 
audiences are higher-level and supervisory-level managers, union 
officials, outside funders, the administration of the education 
providing institution, or others who have invested financial resources 
in the program and who presumably expect the program to achieve 
its stated goals. Evaluations in such cases gather information to 
clarify what stakeholders hope the program will achieve and then 
demonstrate what the program has in fact accomplished ~ in terms 
of anticipated (and possibly unanticipated) outcomes. In turn, it is 
typically hoped that those who control important resources will 
support the program further (Philippi, 1992). 
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Team evaluations attempt to not only produce information 
which stakeholders can make decisions with, but ~ by evolving 
stakeholders in the process ~ generate understanding of and support 
for the program among those stakeholders, as well. Team evaluation 
is also seen as a way of helping stakeholders to develop the team 
decision-making skills they need to deal with other issues they face 
inside the workplace (Sperazi and Jurmo, 1994). 

More-systematic evaluations also are sometimes aimed at 
Others in the larger workplace education field for the purpose of 
demonstrating positive (and sometimes not-so-positive) lessons 
learned in the program. It is hoped that the field will thereby learn 
from the program's experience and improve practice accordingly 
(Kutner, Sherman, Webb, & Fisher, 1991; U.S. Department of 
Education, May 1992). 

More-formal evaluations can also serve instructors and 
learners by clarifying more precisely what in other stakeholders' 
view is being accomplished and what needs to be done to improve 
the program. 

Other potential audiences for evaluations include (Jurmo, May 
1993): 

• Potential participants in the program : They might like to know 
whether they, too, might benefit from participating. These might 
include employees who had signed up previously but left without 
completing the program. 

• Participants' families and communities : They have something to 
gain from programs which increase the knowledge, self-esteem, 
and skills of their members. For example, better-educated 
workers can help their children succeed in school and can inform 
themselves about local issues and get involved in community 
activities. 

• The company's clientele : Customers expect high-quality service 
and might want to see evidence of the company's investment in 
worker training. 

• Re gulatory and certifying agencies : Government (e.g., 
Occupational Safety and Health Administration) and industry (e.g., 
ISO 9000) agencies which certify the quality or safety of a 
company's operations might want to see evidence that the 
company is training workers to handle new demands. 
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• Other companies in the area or in the sam e industry: Other 
companies wondering what to do vis-a-vis worker basic skills 
might like to know what an investment in worker education might 
produce. 

• The company's insurers : Insurers might like to know what the 
company is doing to ensure the safety of its workforce, 
customers, and products. 

• Taxpayers : Taxpayers might want to know how public funds are 
being used in worker education programs. 

• Economic development policy makers : Policy makers in the 
public sector (e.g., government workforce development agencies) 
or private sector (e.g., policy think tanks) might want to know 
where best to invest training funds. 

• News media: News media responsible for covering workforce 
and education issues should welcome information about promising 
developments and emerging issues in workplace education. 

• Vendors : Vendors of workplace education services and products 
should, ideally, develop products based on an understanding of 
what is actually useful in workplace settings. 

The information gathered 

In informal internal monitoring, information is gathered on 
such questions as "how are learners doing?" "what are they 
learning?" "what are they most interested in?" "how are they using 
what they're learning outside the classroom?" Feedback is also 
solicited on how various program components (e.g., curriculum 
content, scheduling, facilities) might be improved. 

More-formal evaluations - both "external" and "internal" ones 
~ look at essentially the same questions. They, however, attempt to 
collect the information from a broader range of sources; they sort 
and summarize it in a more systematic way; they present the 
information to stakeholders who aren't normally so directly involved 
in the program; and they follow up with those audiences to clarify 
any questions they might have and negcdate possible additional 
support from them. 

Several researchers (Barker, 1991; Mikulecky and Lloyd, 1993; 
Philippi, 1992) suggest that evaluations organize information around 
the following questions based on the work which Stufflebeam did in 
non-workplace literacy settings: 
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• Do stakeholders understand and share goals? 

• Are resources sufficient to achieve those goals? 

• Are learning processes and materials sufficient? 

• Have goals actually been accomplished? 

Our own experience with workplace education evaluations 
indicates that evaluations might organize collected information 
around the following categories of questions (Jurmo, May 1993): 

1. Goal clarification : What do stakeholders hope the program will 
achieve ~ not only at the beginning but throughout the life of the 
program? 

2. Assessment of impact : What in fact is the program achieving - 
both in terms of anticipated outcomes and unanticipated ones? 

3. Strengths and limitations : What factors are helping or hindering 
progress toward program goals? 

4. Continuation : Should stakeholders continue to invest in the 
program? 

5. Actions needed : If the program is to continue, what actions are 
needed to strengthen it? Apart from the education program, 
should other actions be taken to meet stakeholders' goals? 

The kinds of information sought in an evaluation depend very 
much on the goals which stakeholders have set for the program 
(Taylor, 1991). In other words, you can't determine whether a 
program has met its goals if you aren't clear about what those goals 
are in the first place. 

As is stated in the first chapter of this report, stakeholders 
have a wide range of expectations for workplace education programs, 
although many in the U.S. have adopted the perspective that 
workplace literacy programs by definition should focus on specific 
job-related reading tasks. 

Sticht (1991), for example, argues that because U.S. Department 
of Education programs by definition are to focus on improving 
learner job performance, then evaluations should attempt to gather 
empirical evidence of changes in job-related behavior in which 
literacy plays a role and which the program focuses on. He suggests 
that evaluations of NWLP programs use the categories of program 
components outlined in the Department's Invitation for Proposal as 
the categories of information to focus on. 
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Recent evaluation work, however, has broadened the scope of 
information a workplace education evaluation might collect. 
Mikulecky and Lloyd (1993) have looked at not only participants' 
abilities to perform particular job-related tasks but at their family- 
related literacy behaviors and at such variables as learners' belief 
about themselves as literate, literacy practices (on and of the job), 
the processes the learners use to deal with literacy tasks, and what 
plans the learners have developed vis-a-vis using their literacy 
abilities. 

Alamprese (1993) suggests that evaluations break outcomes 
down into "participant outcomes" (i.e., impact on individuals' literacy 
skills, job performance, motivation/self-esteem, further education, 
job promotion, and functioning in family and community), 
"workplace outcomes" (i.e., productivity, absenteeism, safety 
turnover, grievances, job restructuring, employee participation), and 
"partnership outcomes" (i.e., expansion of partners, changes in 
partners' commitments and relationships, changes in educational 
programs and support services, and institutionalization of workplace 
literacy services). 

Sarmiento (1991 and 1993) recommends that evaluations 
examine not only changes in individual participants but in the 
behavior of the larger organization. For example, as a result of the 
education program, nas the company incorporated ongoing basic 
skills education into the company training and development 
strategy? 

In recent evaluations of programs in the U.S. and Canada, 
Sperazi, Jurmo, and Folinsbee have found that stakeholders have a 
wide range of expectations for programs and that, to provide 
information of program impact which is relevant to those 
stakeholders, the evaluation process should acknowledge and try to 
respond to each of those expectations (Garlick-Griffin, 1993; Jurmo, 
November 20, 1993; Jurmo, November 23, 1993; Jurmo and 
Folinsbee, 1994; Program Planning Consultant Group, 1992; Sperazi 
and Jurmo, 1994). 

Resources allocated 

Informal, internal evaluation is typically a shoe-string activity, 
consisting of hallway conversations and occasional committee 
meetings. In some cases, instructors devote additional time to 



keeping track of learner progress and discussing progress, problems, 
and next steps with learners. This is sometimes covered in budgets 
under "instructor planning time." 

More-formal evaluations can be fairly costly, especially if 
outside evaluators are brought in. Tiiese evaluators might spend 
some time with the prograim, interview various informants and read 
project documents, develop customized assessment instruments, 
analyze resulting data, and prepare and present reports to various 
audiences. Historically, few resources have been given to evaluating 
programs on the belief that resources should go primarily to working 
with learners. This appears to be shifting in recent years within the 
NWLF, as the latest Invitation for Proposal states that programs 
should pay special attention to developing customized evaluation 
procedures. New legislation also has created new National Workforce 
Literacy Strategies grants which would, among other things, test new 
evaluation approaches (U.S. Department of Education, May 1992). 

How and where information is gathered 

In informal, internal evaluation, information on program 
impact and needed improvements is gathered through the following 
activities: 

• Informal and more-formal tests (including standardized tests, 
customized tests focusing on uses of literacy of particular concern 
in the program, cloze tests, etc.); 

• Simulations (problem- solving scenarios in which learners are to 
demonstrate their ability to deal with a particular communication 
task); 

• Observation of learners at work and in the classroom; 

• Interviews, focus groups, questionnaires, and check-lists for 
learners, supervisors, union representatives, co-workers, 
instructors, and other stakeholders; 

• Document review of learner written work and job-related 
records (e.g., production figures). 

More-systematic evaluations use essentially the same 
procedures but might focus on the analysis and reporting of 
statistical data. Although alternative, "qualitative" assessment is 
gaining more recognition among literacy educators, it is still often 
more difficult to report and less valued, especially by others from 
outside the literacy field. 
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In these data-gathering activities, the sources of information 
include both "human" sources like learners, instructors, supervisors, 
union representatives, and other co-workers, and "material" sources 
like classroom and workplace documents which demonstrate 
workers' literacy and/or job performance. 

How information is organized and analyzed 

In many programs, funders and others require instructors and 
administrators to compile basic information on atte- lance and 
learner progress in some kind of periodic reports. In some cases, 
funders have established guidelines for how that information should 
be organized. In others, it is left up to the education staff to 
summarize basic information in whatever formats seem sensible. 

Some practitioners have developed their own methods for 
compiling useful information for learners. An increasingly-popular 
concept is that of portfolios, or folders, in which the learner and the 
instructor compile various kinds of evidence of learner progress. For 
example, a portfolio might include an attendance sheet completed by 
the learner, a list of goals developed by the learner with the 
instructor, a checklist showing what objectives have been met and 
which need more work, and sample written work and tests 
completed by the learner. 

In programs where standardized tests and workbook exercises 
are used, the instructor can simply compile the test scores into an 
aggregate summary of scores and use them for pre- and post- 
comparisons. 

The responsibility for compiling the information gathered in 
these and other data-gathering activities varies, depending on the 
type of evaluation being done, the resources available, and - in 
particular ~ the program's approach to evaluation and planning. In 
informal, internal monitoring of the program, this responsibility can 
be primarily in the hands of the instructor or administrator, or it 
might be shared with learners and other stakeholders. In 
evaluations carried out by an outside evaluator, responsibility for 
data-gathering and -analysis is typically more in the hands of the 
outsider. 
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How the data are organized and analyzed depends largely on 
the type of data gathered and the expertise and orientation of those 
doing the analysis. The result can be information packaged in 
quantified format or summaries of less-easily-quantifiable feedback 
from various information sources. 

In team evaluations, a representative group of learners, 
managers, supervisors, union officials, and education staff analyze 
the data which they or some individuals on the team gather. 

How findings are reported 

Findings from the relatively informal, internal form of 
evaluation can be reported in informal ways, in written reports or 
presentations to various stakeholders in the worksite. More- 
formalized evaluation reports are typically presented in detailed 
written format, with an executive summary, a description of the 
evaluation methodology used, and various appendices summarizing 
data, data sources, and data-gathering instruments. 

These more-formal reports might be presented to intended 
audiences and then discussed in a follow-up meeting or simply 
mailed to funders and other more-distant audiences, without 
necessarily discussing them further. 

In reports intended to inform others in the workplace 
education field about the program, these reports are disseminated 
through clearinghouses (e.g., the ERIC Clearinghouse on Adult, Career, 
and Vocational Education at Ohio State University); in presentations 
at local, state, and national conferences; through mailings to 
interested parties; and through informal in-person and long-distance 
technical assistance. 

How reports are used 

Internal, informal reports appear to have the most chance of 
being used, as they are tailored to present information known to be 
of interest to audiences close by. It appears that many more-formal 
reports are not used as their authors hope. Practitioners complain of 
sending carefully-prepared reports to more-distant audiences (i.e., 
parties not normally directly involved in program activities who 
nonetheless are actual or potential supporters of the program) and 
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never getting any response from those audiences (Evaluation 
Research, 1992; Sperazi and Jurmo, 1994). 



Sources involved in the National Workplace literacy Program 
sometimes complain that there are inadequate mechanisms for 
ensuring that others in the field can get access to and build on the 
findings contained in the end-of-project reports which all NWLP 
programs must submit. 
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Chapter #3 



STRENGTHS AND NEEDED IMPROVEMENTS 

OF 

CURRENT EVALUATION PRACTICE 

A solid evaluation should respect stakeholder confidentiality; 
be seen as a positive, useful tool for making constructive decisions 
throughout the life of the program; and produce clear, reliable 
information about program impact and needed improvements. 
Evaluation should be guided by a spirit of inquiry, lifelong learning, 
and continuous improvement. Those involved need to understand 
not only evaluation itself but the special nature of workplace 
education. Audiences should use evaluation reports to refine funding 
guidelines, professional practice, and program goals. 

Evaluators need, conversely, to avoid overlooking the 
information which stakeholders really want. Evaluations should not 
be an afterthought or seen as "for someone else. " Evaluation should 
not be limited to "a test." Information should not be of dubious 
credibility, unclear, neglected, or unnecessarily negative in tone. 
Valuable findings should be disseminated widely to those who can 
learn from them. Workplace evaluations shouldn't oversimplify the 
relationship between literacy and productivity. Nor should they 
ignore the insights of practitioners, learners, supervisors, and others 
with valuable knowledge. And evaluations should not be given low 
priority and low funding. 

What a good evaluation should look like 

Our own experience and feedback from others in the field tell 
us that an evaluation of a workplace education program should: 

• Respect the confidentiality of all stakeholders. 

• Be viewed as a positive, useful tool to enable stakeholders to 
make decisions and take constructive actions related to the goals 
they've set for the program. The evaluation process is based on 
"values . . . from the workplace community" itself (Philippi, 1992, 
p. 9). 
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• Produce information about program impact and needed 
improvements which is reliable, clear, and relevant to 
stakeholders. Clear, relevant evidence of positive impact enables 
stakeholders to see the value of the program and continue 
investing in it (Philippi, 1992, p. 13). 

• Produce reports which stakeholders actually use (Kinsey, 1978). 

• Encourage all involved to be frank and open in their analysis of 
the program, in the "spirit of inquiry" and "lifelong learning" 
(Sticht, 1991, p. 10), constructive criticism, and continuous 
improvement. Such "responsive" evaluation responds to 
stakeholders' interests, not externally imposed ones (Philippi, 
1992, pp. 8-9). 

• Be carried out by people who understand not only evaluation 
procedures but the workplace education field, as well (S. Imel, 
personal communication, February 18, 1994). 

• Be based on an understanding of good practice in workplace 
contexts, rather than follow assumptions left over from other 
contexts (S. Imel, personal communication, February 18, 1994). 

• Use procedures which are "user-friendly" and not unnecessarily 
obtuse and complicated, to enable a wider range of stakeholders 
to understand and participate as active players in the evaluation 
process (Kinsey,1978). 

• Be integrated into the program planning process from beginning 
to end (Evaluation Research, 1992; Sticht, 1991), rather than be 
seen as something to be "tacked on" at the end to please a f under. 

• Use criteria relevant to the goals stakeholders set for the 
program (Evaluation Research, 1992). 

• Have enough material and human resources to produce useful, 
high quality results (Evaluation Research, 1992). 

• Uses several data-gathering activities and several information 
sources, to provide information which is useful to all stakeholders, 
to enable them to better understand and support the program. 
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• Be used to shape funding guidelines and practice in relevant, 
useful ways. 

• Clarify what programs should be aiming at and show them how 
to get there, rather than reinforcing inappropriate goals and 
practice. 



What we need to avoid 

We have in many cases learned the above lessons the hard 
way, by being the implemented - or victims - of evaluations in 
which: 

• Confident iality is not respected . In some cases, data are 
gathered or reported in ways which embarrass informants and/or 
those being evaluated. This revelation of information can also 
jeopardize their status in some way. 

• Evaluation is seen as a threat. Those operating a program might 
resist participating in an evaluation because they fear that they 
will be evaluated harshly and their professional and financial 
status jeopardized as a result (Evaluation Research, 1992). 

• Evaluation is seen a s a, hnrrlpn Some might recognize the 
potential benefits of a good evaluation, but feel that the costs of 
staff time, consultant fees, and so forth outweigh the benefits. 
They resist getting involved in evaluation as a result (Evaluation 
Research, 1992). 

• Evaluation is seen as too rompliraterl Some stakeholders might 
recognize the benefit of a good evaluation, but see evaluation as 
something complicated, requiring special expertise th,. t only 
outside experts possess (Mikulecky and Lloyd, 1993). These 
stakeholders feel that they don't have the time, self-confidence, 
and/or interest to develop this expertise and don't want to take 
the time to locate and work with someone who does. 

• Evaluation is seen as irrelevant- Some stakeholders feel that 
they already have a good sense of what is being achieved and 
what needs to be done to strengthen the program. They know 
this from direct experience interacting with and observing staff 
and learners. They feel that it's therefore not worth the extra 
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effort required to set up a formal evaluation which wouldn't likely 
tell them anything they don't already know and which might 
instead produce a distorted, overly-narrow, or irrelevant picture 
of the program. 

• The evaluation misses information importan t to stakeholders. 
Some evaluations might do a good job of gathering particular 
types of information while missing out entirely on other 
information which some stakeholders could benefit from. A 
workplace education evaluation might, for example, do a good job 
of reporting test scores, documenting attendance and the number 
of modules taught, but fail to capture what impact all of this 
activity had on the learners' day-to-day life on or off the job. 
Another example: A learner might want to know - in simple 
terms - "how am I doing?" and "where should I go from here?" 
She might not be interested in an evaluation which produces 
figures on attendance, costs-per-student, and reduced scrap rates. 

• Evaluation is an afterthought . Evaluations too often are begun 
at or near the end of the program, rather than integrated into the 
planning process from the start. In such a case, there is unlikely 
to be any baseline data against which learner outcomes can be 
compared (Sticht, 1991). 

• Evaluation is sppn as "for someone else. " Some think "reports to 
funders" when they hear the word "evaluation." Rather than see 
evaluation as a useful tool for decision-making, they see it as a 
cumbersome ritual to produce information for someone else who 
probably doesn't pay much attention to it anyway. 

• Evaluation is e quated with "a test. " Rather than seeing 
evaluation as process of gathering information and making 
decisions related to program impact and needs, many equate 
evaluation with some kind of "test." It is assumed that this test 
will measure participants' skills and produce a set of numerical 
scores as evidence of what is happening in the program 
(Evaluation Research, 1992). 

• The information generated is of du bious credibility. In some 
cases, evaluations produce information which is "of a suspicious 
nature." Those gathering the information might have unknowingly 
or knowingly asked the wrong questions or asked them in a way 
which failed to elicit honest, accurate information. Or perhaps 
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those who analyzed the information mistakenly or purposely 
distorted the information to produce conclusions not based on the 
evidence gathered. What is lacking is "empirical, confirming 
evidence"; instead there is "too much subjective self-reporting" 
with "a danger of self-deception and therefore bias" (Sticht, 1991, 
p. 10). 

• The information generated is not clear . Sometimes those 
gathering information record too much detail, record it in a 
jumbled or confused way, or otherwise produce information which 
is difficult if not impossible for others to understand. 

• Reports are produced but not used . Programs in some cases go 
to the trouble and expense of producing an evaluation report 
which accurately reflects program impact and needs, but they 
then find that the audiences to whom they submit the report don't 
use it appropriately. Audiences might be too busy or not 
understand the field enough to see the value in the information 
represented in the report. For example, an employer might let a 
report sit on her desk because she is too busy with other business. 
A public funder might be simply overwhelmed with too many 
such reports, let them pile up, and never assign anyone to study 
them and use them to refine funding guidelines. Whatever the 
reason, those getting such a response to a good quality evaluation 
effort are less likely to invest their resources in producing a solid 
one the next time around (Evaluation Research, 1992). 

• Valuable evaluation findings aren't shared with audiences which 
might be interested in them . Sometimes evaluations produce 
information which would be valuable to particular audiences but 
the findings are never shared with those audiences. For example, 
potential program participants might like to hear how the 
program has helped their co-workers, but no one takes the time to 
present that information to them in a clear way. Similarly, other 
businesses or unions in the community might like to hear how a 
successful workplace education program operates, but no one 
thinks about providing such information in such forums as a 
Chamber of Commerce luncheon or coverage in the local 
newspaper. Perhaps most troubling is the fact that there are few 
professional development opportunities for practitioners in which 
they could get access to and learn from those evaluation reports 
which have been done. 
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• Discussions of problems have a "negative" tone . Some programs 
operate in a context where there is a history of poor relations 
among some stakeholders. In such a context, it is difficult to 
present evaluation as a way of identifying both positive 
achievements and problems to be constructively dealt with. Focus 
groups can degenerate into "gripe" sessions rather than an 
opportunity to continuously improve the education program. 

• Evaluations are carried out by people who don't adequate ly 
understand the field . The workplace literacy field is relatively 
new, and few have had experience doing evaluations in this 
unique and evolving context. Evaluations might thus be put in the 
hands of people either with little or no prior evaluation 
experience or who have done evaluations in contexts much 
different from the program to be studied. This increases the 
likelihood that the evaluation will focus on the wrong kinds of 
questions or be guided by assumptions and standards not 
appropriate to adult workers in today's changing workplaces 
(Evaluation Research, 1992; S. Imel, personal communication, 
February 18, 1994). 

• Evaluations over-simplify the relationship between literacy and 
productivity . The relationship between a worker's basic skills and 
his or her job performance is not straightforward. Many other 
factors can impact on the worker's job performance, so an 
evaluation which assumes that an improvement of a participant's 
job performance can be attributed to participation in the basic 
skills program might be misleading (Evaluation Research, 1992; 
Sticht, 1991). 

• Evaluations define focal points narrowly , when broader and 
more complex information needs to be examined. This has the 
effect of reinforcing inappropriate assumptions about workplace 
education. Some of these arguments for new ways of looking at 
programs are outlined below: 

Schultz (1992) says that rather than defining literacy education 
as "reading and writing, and possibly listening and speaking skills 
handed to tl 2m by their instructors," programs should view 
"literacy and learning as active, constructive processes which 
build on the knowledge teachers and learners bring with them" 
(p.14), a perspective "more appropriate to the new workplace" (p. 
iv). (She acknowledges, however, that the demands which 
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workers will have to contend with in the new workplace have not 
yet been adequately identified.) 

Gowen (1990) argues that evaluations which focus 
quantitatively on increased productivity, improved attendance, 
and increased discrete basic skills fail to provide information 
about three important areas: "the literacy beliefs that inform the 
actions of adults perceived as in need of enhanced literacy skills; 
the literacy beliefs of the providers who fund and develop 
curriculum and pedagogy; and the social and historical contexts 
and institutional structures that foreground and help shape these 
belief systems" (p.3). 

Stein (May 1993) cites researcher Sylvia Scribner's view that 
workplace literacy research needs to re-think how learning is 
viewed in a workplace context: "School-based research has 
fostered a conception of learning as an activity separate from 
other life activities. But as soon as we enter the workplace, we 
find that this conception does not hold: the defining feature of on- 
the-job training is that teaching and learning are occurring 
simultaneously with getting-the-job-done, with working. Training 
occurs in the course of working ... It is embedded in on-going 
work activities" (pp. 5-6). 

Mikulecky and Lloyd ( 1993) cite Ly tie's suggestion that 
"performance measures (tests and exercises) miss a good deal of 
important information about adult literacy learning" (p. 30). They 
have broadened the scope of their evaluation work beyond just 
looking at productivity-related measures to look at learners' 
beliefs about literacy and themselves, learner literacy practices, 
the literacy processes employed by a learner while reading, and 
the plans a learner has which may involve literacy use. 

Mikulecky and Lloyd also acknowledge that workplace 
education programs might encompass a number of goals beyond 
enhancing individual workers* ability to read job-related 
materials. A "multi-strand" program might also include GED 
classes, an ESL course, home study or self-paced computer work 
on materials not necessarily job-related, and technical training 
(pp. 3-4). Rather than being seen as distractions from job-related 
goals, these services can reinforce each other by providing 
multiple opportunities and incentives for learning (p.ll). They 
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also might be incorporated into an evaluation of the overall 
program. 

Schultz (1992) and Rosen (Sperazi, Jurmo, and Rosen,1991) 
argue that evaluations should define curriculum as a process 
rather than as a product. Those that focus on numbers of modules 
and contact hours miss what the curriculum really consists of. 

Sarmiento (1991) suggests that evaluations look at not just 
impact on individual workers' behavior but at changes in the host 
organization, as well. That is, if employers don't provide 
incentives and opportunities for workers to use what they are 
learning in the classroom, an education program will have limited 
lasting impact. 

Hull (1991) argues that 

... we have not paid enough attention, as we measure 
reading rates, design curricula, and construct lists of 
essential skills, to how people experience instructional 
programs and to how they accomplish work . . . Nor is it 
common, in studies of work or reading and writing at work, 
to acknowledge the perspectives of workers - to discover 
the incentives and disincentives they perceive and 
experience for acquiring and exercising literate skills. 
Alternate points of view and critical reassessments are 
essential if we are ever to create frameworks for 
understanding literacy in relation to work; if we are ever to 
design literacy programs that have a prayer of speaking to 
the needs and aspirations of workers as well as employers; 
and most importantly, if we are ever to create structures for 
participation in education and work that are equitable and 
democratic ... we have got to let some different voices be 
heard (p. 2). 

Those with significant knowledge o f the program are not given 
o pportunities to document and reflect on what thev have le arned, 
Schultz (1992) calls for teacher-research, to enable "teachers to 
explore their own guiding philosophies and varied ways of 
interacting with learners." Such descriptions "could provide 
valuable information to program planners and policymakers." She 
suggests that learners also be invited "to participate in research 
about themselves and their own learning" (p. 50) to clarify the 
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knowledge and practices they bring with them, how they go about 
learning on and off the job, and other important questions not 
normally considered. 

Pritz and Imel ( 1993) similarly call for analysis by 
practitioners "in the trenches" about the "innovative practices that 
arise spontaneously as part of an activity, as well as the tacit 
knowledge practitioners possess about their work." The field can 
merge this "craft knowledge" — "insights and valuable ideas" — 
with other "scientific knowledge generated by researchers and 
evaluators" (p. 119). 

• Evaluation is given low priority and low funding : Money- 
starved workplace educators have understandably channelled 
limited available resources into instruction rather than evaluation, 
which is seen as almost a luxury or as an academic exercise. For 
this and many of the other reasons cited above, evaluation is 
given low priority and, therefore, few resources. Even evaluators 
who recognize the above evaluation-related needs and want to 
develop more appropriate forms of evaluation have to try to do so 
with small budgets. This discourages experimentation and 
reinforces the status quo of both evaluation and workplace 
education. 
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Chapter 4 



PROMISING DEVELOPMENTS 



Innovators in the workplace education field are developing 
new ways of evaluating workplace education programs and, with 
them, new forms of workplace education itself. Many of these 
innovations emphasize collaborative decision-making by 
stakeholders. Education programs are thereby more directly tied in 
with new ways of organizing work which emphasize teamwork and 
continuous improvement. 

Within the workplace education field, some funders are now 
encouraging programs to follow collaborative decision-making 
principles. Practitioners are coordinating all program components in 
an integrated cycle characterized by continuous, collaborative 
decision-making. Stakeholders are being encouraged to first analyze 
the organization's overall human resource needs and policies before 
deciding whether to embark on an employee education initiative. 
Practitioners are creating new forms of individual assessment, 
curriculum development, and evaluation which put greater 
responsibility into the hands of program participants and other 
stakeholders. Practitioners are applying collaborative learning 
principles to their own professional development, by establishing 
peer-support systems. 

In the related fields of adult literacy education, social sciences, 
and training and organizational development, innovators have 
developed assessment, evaluation, and program models which mirror 
many of the innovations being developed by workplace literacy 
educators. 

Each of these disciplines might benefit from greater cross- 
fertilization with the others. 



In the previous chapter, the reader is presented with a lengthy 
"to do" list. It describes actions which should be taken to create 
meaningful evaluations. 

In fact, many practitioners, researchers, employers, unions, and 
funders are trying to develop new ways of thinking about ~ and 
planning and implementing -- evaluations of workplace education 
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programs. Many of these innovators see evaluation as a tool for 
creating the kinds of collaborative workplace education programs ~ 
and collaborative, learning-oriented workplaces ~ described in 
Chapter I. This vision is important, as it provides motivation for 
work which is often unrecognized and minimally funded. 

Listed below is a sampling of such efforts in the workplace 
education field, along with evaluation and education practices being 
developed in such related disciplines as adult literacy education, 
evaluation, organizational development, and training. These 
examples are presented for those looking for ideas and inspiration to 
use in their own efforts to improve evaluation practice and policy. 



Within the workplace education field 
Changing funding guidelines 

The Massachusetts Workplace Education Initiative : In 1991, the 
Massachusetts Workplace Education Initiative (MWEI) changed the 
way it approached evaluation of the workplace programs it funded. 
For three years, the Initiative had relied on an outside evaluator and 
occasional updates from program directors about attendance, test 
scores, and so forth. Based on recommendations from its evaluation 
consultant, Evaluation Research, MWEI agree to explore new ways of 
more actively involving management, union representatives, and 
learners in goal-setting and monitoring of their programs. The 
Initiative also wanted programs to produce clearer evidence of 
program progress toward stakeholder goals. 

MWEI asked its outside evaluator, Evaluation Research, to 
develop a new approach to evaluation which would respond to those 
guidelines. The result was a pilot project in which six workplaces 
created evaluation teams whose members (1) clarified and 
prioritized goals, (2) identified what information they wanted from 
an evaluation and how they would use it, (3) designed and then used 
data-gathering activities to collect the desired information, (4) 
analyzed the resulting information, (5) reported their findings to 
appropriate audiences, and (6) took follow-up action in response to 
the report itself and feedback from their audiences (Sperazi, Jurmo, 
and Rosen, 1991). 
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MWEI felt that this pilot project demonstrated the value of a 
systematic, "team" approach to program planning and evaluation. 
During the next few years, guidelines were revised to require all 
funded programs to be managed by teams composed of 
representative stakeholders. The Initiative provided training and 
technical assistance to those teams, to help them perform the new 
roles required (Curry, 1993; Sperazi, 1991). 

The U.S. Department of Education : The U.S. Department of Education 
has generally strictly adhered to the "functional context" perspective 
described in Chapter I. In recent years, however, practitioners and 
researchers have questioned the Department's particular 
interpretation of the concept of contextualization and the legislation 
which supports the National Workplace Literacy Program. 

For example, in September 1991 the Department called 
together representatives of 39 NWLP projects and asked them for 
feedback on six issues: establishing and maintaining partnerships, 
curriculum, recruitment, assessment and evaluation, worker 
involvement, and future policy directions. In a constructive, positive, 
two-day meeting, participants suggested many ways of 
strengthening all of these program components. Many participants 
asked that NWLP not require programs to focus on narrow job skills 
which research and experience were showing don't reflect what 
workers and companies actually need and are interested in. Many 
also asked for new ways of involving all stakeholders in program 
planning, implementation, and evaluation. Some participants also 
suggested that NWLP fund only programs clearly linked to a high- 
performance model of organizing work. These requests 
demonstrated that the NWLP was fulfilling its mission of allowing 
practitioners to identify what was needed to provide effective 
workplace educaiton services. In this case, practitioners were asking 
that the NWLP continue to encourage creative innovations in 
response to emerging workplace needs rather than require programs 
to go down a single path predetermined several years earlier 
(Evaluation Research, 1992). 

Subsequently, the Department's particular way of interpreting 
"contextualization" was more pointedly questioned in several widely- 
circulated papers. Schultz (1992) argued that NWLP had latched 
onto a "new orthodoxy" which defined functional literacy in a way 
which doesn't reflect good literacy practice or what the new 
workplace actually requires of workers. Sarmiento (1993) claimed 
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that NWLP was pushing programs to reinforce outmoded, Tayloristic 
roles for workers rather than dynamic, active, thinking ones. He said 
that NWLP was interpreting legislative guidelines too narrowly. 
Gowen (1992) said that the NWLP project she worked in was 
essentially irrelevant, in that workers were given little say in what 
they were to learn and the curriculum focused on topics of little 
interest or use to them. Hull (1991) claimed that the functional 
context model tended to isolate workers' basic skills from other 
factors affecting their productivity and isolate workers' themselves 
from the process of setting workplace education goals. The result is 
programs which miss the mark in terms of helping workers and their 
organizations make relevant improvements. ' 

Perhaps partially in response to the above arguments, the 
Department has adopted some of the thinking which is basic to the 
collaborative approach. For example, it has now acknowledged the 
need to more actively involve all stakeholders (including learners) in 
program planning and governance, and to invest more in ongoing 
evaluation (U.S. Department of Education, May 1992). It has also 
stated that: 

. . . the future of workplace literacy (is linked) to the American 
economy's shift from traditional production organizations to 
high performance organizations. In traditional organizations - 
and in traditional workplace literacy programs ~ workforce 
learning is not viewed as a meaningful activity in relation to 
the production process. Traditional workplace programs are 
short-term and problem-centered. High-performance 
organizations view workplace education as an integral aspect of 
the production process. As the American economy transitions 
to the twenty-first century, workplace education must respond 
to the education needs of high performance organizations (p. 
11). 

This statement borrows the language of advocates calling for 
more-collaborative, team-based approach. It can be contrasted 
with other statements made by the Department during the same 
period, which claimed that the Department's adherence to a 
stricter functional context approach was required by 
Congressional statute (Evaluation Research, 1992, p. 47 and 
Appendix). The fact that the Department was using the language 
of "liigh-performance" advocates was taken by some as a sign that 
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the Department was willing to consider new ways of defining and 
responding to workplace education needs. 

Creating new program models 

Largely in response to their own experience and to the 
arguments by proponents of a collaborative approach, practitioners 
are developing new collaborative models and practices. Some 
examples are outlined below: 

Integrating program components into a collaborativ e 
planning ryHp- ABC CANADA (based in Toronto), with Literacy 
Partnerships (based in New Jersey), has mapped out a 
collaborative model for planning workplace initiatives which 
integrates various program components into a cyclical (rather 
than linear) arrangement. From its earliest stages, the process 
relies heavily on collaborative decision-making by a team of 
stakeholders. The team first carries out a broader analysis of the 
organization's needs before focusing on employee basic skills per 
se. An evaluation plan is designed prior to implementation of 
program activities, and results are continually fed back to the 
team for continuous improvement. This model resembles and is 
designed to support other team-based continuous improvement 
activities being used in the organization (Folinsbee and Jurmo, 
1994a; Jurmo, April 1994; Waugh Folinsbee, 1992; Waugh 
Folinsbee, 1991). 

Analyzing org anizati onal needs : ABC CANADA has pioneered 
a method of analyzing employee basic skills needs as one of many 
needs a company should consider. Called a "workplace needs 
assessment" (or "organizational needs assessment"), this process is 
carried out by a site planning team like those developed in the 
Massachusetts Workplace Education Initiative, above. This 
analysis is the first activity undertaken by a team. Members 
examine the key factors which impact an employees' job 
performance and quality of work life, one of which might be basic 
skills. The analysis identifies which of those factors impede 
workers and what steps the organization can take to remove those 
impediments. This collaborative process demonstrates that an 
employee education program will not, by itself, help an 
organization transform itself to a more productive and just 
workplace. Rather, both the organization and individuals must 
take steps to continually improve themselves. 
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This process is broader in scope than the literacy task analysis 
process commonly used in functional context programs. It does 
not automatically assume that employee basic skills is the 
primary problem to be dealt with. Rather, the planning team goes 
in with an open mind, trying to figure out what a company must 
do to assure that workers can use the skills they already have and 
might develop in an education program. By starting the planning 
process with a team, an organizational change initiative builds 
stakeholder understanding of and buy-in for the program from 
the start. Once the organizational needs analysis is done, the team 
continues to oversee other components of the planning cycle 
(Folinsbee and Jurmo, 1994b; Waugh Folinsbee, 1992; Waugh 
Folinsbee, 1991). 

In the United States, this methodology has been adopted by 
REACH , an education program for New York State employees. 
Sponsored jointly by the Governor's Office of Employee Relations 
and the Civil Service Employees Association, REACH staff have 
conducted such organizational needs analyses at several sites in 
the past year. They have thereby not only determined needs at 
those sites but clarified how such a needs analysis process might 
be adapted in other sites around the state (Jurmo, April 25,1994; 
Sperazi and Jurmo, 1994b; Medina, April 28,1993; Young, 1993). 

At least five other workplace education providers - the 
University of Tennessee's Center for Literacy Studies, the New 
York State Education Department, LaGuardia Community College, 
the Suffolk County (NY) BOCES, and the Massachusetts Workplace 
Education Initiative -- have now adopted the workplace needs 
assessment concept and built it into proposals submitted to the 
NWLP in the spring of 1994. 

Analyzing individuals' needs, interests, and abilities : 
Some practitioners are borrowing from the reading field ~ 
particularly the whole language perspective ~ and redefining 
what "individual needs assessment" should be assessing and how 
to go about doing so. Rather than see workplace literacy programs 
solely as a way of changing learners' work-related behavior, these 
practitioners argue that programs should be concerned with 
deeper learner beliefs and behaviors vis-a-vis literacy and 
learning. Otherwise, a learner might be able to master a few 
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discrete job tasks but not develop a desire and ability to continue 
learning and applying literacy in other aspects of her life. 

Mikulecky and Lloyd (1993), for example, have borrowed from 
researcher Susan Lytle to study "learner beliefs about literacy 
and themselves, learner literacy practices, the literacy processes 
employed by a learner while reading, and the plans a learner has 
which may involve literacy use" (p. 30). This is a major departure 
from functional context advocates' normal focus on job-related 
measures. 

The Workforce Instructional Network , a NWLP project 
operating out of Southwest Texas State University in San Marcos 
in the early 1990s, developed a "collaborative portfolio approach" 
to "qualitative assessment." The Network argued that qualitative 
measures are a more valid way of helping fit curriculum to actual 
learner needs and of mapping learner growth for learners and 
other audiences than are traditional standardized tests. Rather 
than rely on numerical test scores, qualitative procedures 

. . . provide rich and detailed information about the learner's 
literacy abilities. Multi-dimensional in nature, the 
qualitative assessments attempt to provide a holistic picture 
of . . . abilities from many angles. They are designed to be 
more authentic — more like the literacy tasks learners 
actually face ~ than traditional tests . . . Since the 
qualitative asessments can be used flexibly to fit the 
learning situation, they allow instruction and assessment to 
be more closely linked. When used as self-assessment tools, 
they can help the learner develop metacognitive abilities 
that can lead to greater learning independence . . .(Johnson 
and McBride, 1993, p. 1). 

A learner's portfolio would include both reading and writing 
scales. In the reading scale, the learner provides 

... a written or oral re- telling of (a chosen) text, . . . 
underlines main ideas and details within the text itself, and . 
. . answers literal, inferential, and application level 
questions. This variety of tasks provides multiple measures 
of that slippery construct, "reading comprehension," and 
thus aids in ascertaining some knowledge of a student's 
processing strategies (p. 1). 
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The Network's writing scales quantify writing behaviors (e.g., 
planning what to w^ite, revising of one's own writing), products of 
each step of the process (e.g., originality, organization of the 
original outline), and the final product of the process (e.g., 
content, organization, and mechanics of the final draft). 

The NWLP project at El Paso Community College (EPCC) has put 
considerable effort into developing a mix of assessment tools. 
Special effort has been made to create "seamless assessment," 
assessment integrated with instruction in a way which is not 
obtrusive and is regarded by the learner as another learning 
activity rather than a "test." 

EPCC has experimented with a large number of data-gathering 
activities designed to (1) assess whether learners are developing 
the language strategies taught in the curriculum (via customized, 
"authentic/holistic" tools), (2) assess whether learners are making 
gains in "general" (non-curriculum-specific) skills (with a 
standardized test, (3) assess whether learners are transferring 
their skills to job-related tasks (via questionnaires and interviews 
with various stakeholders), and (4) get feedback from 
stakeholders about the course (via questionnaires and interviews). 

After four years of developing, using, and refining various 
assessment and evaluation tools, a newly-reorganized staff 
evaluated these data-gathering activities in early 1994. The staff 
agreed that, if they were going to be promoting the notion of 
"Total Quality Management" to the companies they were working 
with, then the project itself should be managed in a TQM, high 
performance mode. They agreed that a key element of a high 
performance organization is "continuous improvement" through 
ongoing monitoring of progress and program needs. They decided 
that such monitoring would be done through the following 
activities: 

Assessment of whether learners are achieving learning 
objectives imbedded in curriculum : Done through: 

• Teacher observation sheets through which instructors 
observe learners in group tasks and rate their oral 
production and group process skills. 
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• Interactive check sheets in which learner and instructor 
monitor reading fluency, vocabulary, and strategies. 

• Holistic writing scales in which instructors rate learners' 
progress according to several criteria. 

• Learner self-assessment in which learners assess their 
abilities to perform selected job-related tasks. 

• Dialogue journals in which learners describe how valuable 
the lesson is, how they are using what is taught, suggestions 
for future topics, etc. 

• Videotaped interviews in which learners are taped using 
English at the beginning of the program for comparison to 
similar tapes at the end. 

• Review of learner writings in which instructors rate 
learners' abilities to write about the topics discussed, make 
predictions about the possible results of a particular action, 
etc. 



Assessment of whether learners are applying what they have 
learned to contextual tasks : Done through: 

• Interviews with learners and management 
representatives. 

• Review of job-related records. 

Clarification of actions needed to improve the program : Done 
through: 

• Interviews and focus groups with various stakeholders. 

• Staff meetings. 

Clarification of learning objectives : Done through: 

• Interviews and focus groups with various stakeholders. 

• Dialogue journals and other communications between 
participants and instructors. 

(Jurmo, November 23, 1993, p. 16; Jurmo, May 1994). 

REACH has contracted with the City University of New York 
(CUNY) to provide basic skills services to employees in several 
state-operated facilities in New York City. Goals for the program 
vary from site to site and worker to worker, and typically include 
a mix of job-related and personal-development objectives. There 
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is a general consensus among stakeholders - including 
management — that the program should provide a "richer" kind of 
education, to challenge learners to engage in ongoing learning 
activities beyond focusing on skills required in particular job 
tasks. 

CUNY staff recognize that, to be useful, instruction needs to 
focus on real needs. To clarify those needs, staff have developed a 
number of authentic assessment tools to identify learner interests 
and abilities. For example, an instructor brings in a sampling of 
books which she feels — from prior experience ~ that learners 
might have an interest in. (These include novels, biographies, 
history, and civil service practice books.) In an interview, the 
instructor asks the learner what kinds of materials he or she 
currently reads or might like to read, to get a sense of learner 
interests and behaviors. 

The instructor might then give the learner a Gateway exam, in 
which the learner reads a particular passage until she reaches a 
point where she has trouble understanding it. 

The instructor might then ask the learner to provide a writing 
sample. In the writing, the learner might describe what she has 
read, describe something about herself and her goals, or write an 
essay on one of three possible topics. 

As instruction gets underway, the learner might use a checklist 
containing the topics which she and the instructor have agreed to 
cover. As she deals with each topic, the learner checks off 
whether the topic has been covered, not covered, or needs more 
review. 

Learners store data from these various exercises in personal 
portfolios. The instructor has access to these portfolios and can in 
aggregate form summarize information for the entire group of 
learners. This summary can be presented to other stakeholders, 
as evidence of learner progress (Jurmo, April 25, 1994). 

At Ohio State University , the Center on Education and Training 
for Employment has developed an approach to identifying 
workplace literacy needs which attempts to avoid the limitations 
of a traditional literacy task analysis. With the guidance of a 
facilitator, a selected group of workers from a particular job 
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classification spends several days systematically analyzing then- 
jobs to identify tasks performed, along with the knowledge, skills, 
and attitudes necessary to perform them. The team focuses in 
particular on the reading, writing, speaking, listening, 
computation, problem-solving, and decision-making skills 
required. The outcome of this process is a detailed analysis of the 
job along with enhanced self-confidence and analytical and team 
skills for team members. (The team's analysis is further validated 
and refined by other employees to whom it is shown.) 
Participating workers also emerge with an understanding of and 
sense of ownership for the education program which follows (Pritz 
and Imel, 1993). 

Ottawa-based researcher Glenda Lewe has been a pioneer in 
literacy task analysis (LTA). She became concerned, however, 
that too often LTAs focused on identifying supposed worker 
deficiencies and — in some cases ~ those "deficiencies" were 
reported to employers. She concluded that LTAs can be done in 
two ways: one is structured and deficiency-oriented and the other 
is more flexible and (at least potentially) empowering. 

She has more recently focused on developing the latter 
approach to LTA. She focuses on "strong" workers — those 
commonly seen as already doing their jobs well. She analyzes 
what is needed to do the job rather than looking for who is 
"deficient." 

She focuses on a cluster of jobs in which there is a possible 
path to promotion. For example, she might look at a cluster of 
carpentry jobs: carpenter apprentice, carpenter, and foreman. 
Within that cluster, she examines the incremental skills needed to 
go from one job to the next. Such an analysis identifies learning 
objectives workers might focus on to qualify for promotion (Lewe, 
1993; G. Lewe, personal communication, April 8, 1994). 

In 1993-94, REACH (with funding from the National Institute 
for Literacy) set out to develop a methodology for assessing the 
literacy needs of a range of workers (cleaners and custodians, 
nurses assistants, clerks, aides in a center for the developmentally 
disabled) in six New York worksites. Through ethnographic 
observation and interviews, the research team found that each 
site had its own culture and communication patterns. 
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The researchers also learned that most workers could already 
perform their current jobs satisfactorily and took pride in their 
work. Most of the workers also faced little paperwork in their 
jobs. 

There were, however, some factors which might require 
workers to strengthen their basic skills. Some ~ but not all 
workers wanted to be able to pass civil service exams for 
advancement. Some jobs were also being redesigned to require 
workers to take broader responsibilities and do more paperwork. 
And some sites were introducing "quality" initiatives requiring 
employees to speak up in team meetings and handle conflicts. 

The researchers concluded that, to understand individual 
workers' needs, one must first understand the larger 
organizational context. To do so, the kind of upfront workplace 
(organizational) needs assessment described above (under 
"Analyzing organizational needs") is needed. Such a process, the 
researchers felt, would help all stakeholders clarify organizational 
and social factors which might support or inhibit learning and job 
performance. If this analysis indicated that a basic skills program 
might be needed, at that point individual needs, interests, and 
opportunities could be assessed ( C. Young, personal 
communications,1993-94; Jurmo, April 25, 1994; Young, 1994). 

Curriculum desig n: A number of programs are trying to create 
the collaborative learning model called for in Chapter I. These 
programs generally adopt the collaborative approaches to 
analyzing organizational and individual needs described here, 
along with team evaluation procedures. 

In terms of developing collaborative instruction integrated 
with team management, only a few programs have gotten very 
far. Stein (1993) describes one program at United Electric 
Controls in Massachusetts in which the instructor trains program 
participants in how to serve as facilitators of learning and 
problem-solving activities back on the shop floor. Learning is 
thereby moved out of the classroom and into the hands of 
workers on the shop floor. 

Jurmo (in press) describes another case in which a newly-hired 
instructor found that the functional context curriculum he was 
handed was essentially irrelevant. The curriculum had been 
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designed more than a year earlier by outside consultants, and in 
the intervening period many of the job tasks imbedded in the 
curriculum had been eliminated from the company. The tasks 
covered in the curriculum which still existed in the company, the 
instructor learned, were for most participants irrelevant: the 
employees already had an acceptable mastery of those tasks 
and/or they were tasks which they simply were not interested in 
focusing on. 

After trying to run the course for two weeks, the instructor 
decided to throw out most of the existing lesson plans and 
reconfigure the class as a decision-making team. He hoped that 
this would make sense to the learners and to the company, given 
the fact that the company was promoting many of the 
collaborative decision-making functions of "TQM." He asked 
participants what skills and knowledge they wanted to develop. 
They told him they wanted to know how to qualify for better jobs 
(within and outside the company) and how to further their 
education. 

The instructor decided to re-organize the remaining eight 
weeks of sessions into team decision-making activities. Learners 
analyzed their own jobs and those of their supervisors, to identify 
what tasks and skills were required. They analyzed their own 
existing skills and knowledge, identifying what they had learned 
in their schooling, home life, hobbies, and jobs. They then 
summarized that information in resumes. They also identified 
and then analyzed particular problems they faced on their jobs 
(e.g., an inefficient freight elevator system) and mapped out action 
plans specifying problems, causes, and possible corrective actions. 

The instructor emerged from this project with the sense that 
such a collaborative learning and decision-making process had 
great potential for the development of individual workers and 
their organization. Participants learned how to identify problems 
and learning goals and to analyze what was needed to solve those 
problems and meet those goals. In the process, they developed 
research, teamwork, and oral and written communication skills. 
They also produced concrete products (e.g., resumes and work- 
related action plans) which they could use to improve their own 
situations and workplace operations. 




The Center for Literacy Studies at the University of Tennessee, 
the New York State Education Department. ABC CANADA, and 
LaGuardia Community College fin New York City ) have recently 
developed proposals for similar team-learning projects integrated 
with continuous improvement and learning organization 
principles. 

Evaluation : As stated under "Changing funding guidelines" 
above, the Massachusetts Workplace Education Initiative — 
through its evaluation consultant Evaluation Research - 
pioneered the development of team evaluation. The methodology 
first tried there has since been further refined in projects funded 
by the National Institute for Literacy in the U.S. and by ABC 
CANADA in that country. (This report is one of the products of the 
NIFL-funded project.) In both of these more-recent projects, the 
team evaluation process has been more deliberately linked to 
other program planning components and to the growing shift 
toward team management and learning organization models. 

The Center for Working Life in Oakland, California developed a 
"participatory evaluation" model for the "worker-centered" 
education program it helped design in a Montana sawmill. 
Instructors and "peer counselors" ~ selected program participants 
who served as liaisons between fellow participants and other 
stakeholders ~ were trained in worker-centered principles and 
practices. They then developed a strategy and tools for gathering 
data to document program progress and needed improvements. 
The evaluation team interviewed all stakeholders, including 
teachers, several times during the program. Staff members were 
also asked to keep journals in which they wrote wecidy 
summaries of high and low points, problems and solutions, and 
lessons learned. Evaluation team members were also available for 
consultation with staff on an as-needed basis. The resulting 
"feedback loop" fed information to education staff and other 
members of the multi-party Partnership Council which oversaw 
the program. Data were gathered via interviews, questionnaires, 
and meetings for various stakeholders, as well as from special 
journals kept by instructors (Atkin, 1993; L. Shore, personal 
communication, April 8, 1994). 

In 199?, the New York State Education Department sponsored a 
series of workshops in which staff of its funded workplace 
education programs were exposed to the team evaluation model 
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developed the previous year in Massachusetts. The coordinator of 
the NYSED workplace initiative hoped programs would use this 
methodology to generate more involvement of stakeholders at the 
site level. He also hoped that sites would produce evaluation 
reports which were more formative in nature -- detailing actions 
they planned to take to strengthen their programs ~ rather than 
merely reporting attendance figures and test scores (Jurmo, March 
1992; Jurmo, June 1993). 

Staff from REACH (the education program for state workers 
described above under "Analyzing organizational needs") were 
among the participants in those New York State workshops. They 
eventually adopted the team notion in a major way. REACH staff 
felt that such an approach would provide the infrastructure of . 
stakeholder support which programs need. REACH called the 
building of stakeholder support "site development," something as 
vitally important as good "curriculum development." In 1993 and 
1994, REACH conducted team evaluations at two of its sites (I. 
Baumgarten, personal communication, June 21, 1994; Jurmo, April 
25, 1994; Medina, October 7, 1993; Medina, February 18, 1994; 
Sperazi and Jurmo, 1994b). 

As noted under "Changing funding guidelines" above, the 
Massachusetts Workplace Education Initiative has since 1991 
encouraged its funded programs to take a team approach to 
planning and evaluation. Most recently, the Initiative received 
evaluation reports from seventeen site teams, an indication of the 
viability of this concept (Sperazi, 1994). 

In 1993, the Connectic ut Business and Industry Association 
used the team-based evaluation model developed in 
Massachusetts for the thirteen workplace basic skills programs 
under its direction. Funded through the NWLP, these programs 
are to prepare workers for the new demands of the high 
performance organization model. Evaluation teams in each site 
have been establishing appropriate evaluation criteria and 
gathering some data with the help of an external evaluator and 
the CBIA's coordinator. At present, the external evaluator is 
responsible for much of the data-collection and -analysis, but the 
hope is to involve evaluation teams more actively in all aspects of 
the evaluation process in future funding rounds (L. Sperazi, 
personal communication, May 17, 1994) . 
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Professional development opportunities 

The innovations represented above require professionals with 
particular expertise and vision. Many of those who have taken the 
lead in developing collaborative theory and practice have had to do 
so through experience -- by the seat of their pants and by 
scrounging around for guidance in reference materials, conferences, 
and grapevine contacts. Some of those involved in creating these 
new forms of workplace education have recognized that, if they want 
to share expertise and build an identity among like-minded 
professionals, they will have to create their own staff development 
opportunities. 

The Massachusetts Workplace Education Initiative and the 
Texas Workforce Education Consortium are two statewide networks 
which have intentionally put the collaborative approach on the 
agendas of their annual conferences. Many of the practitioners and 
researchers mentioned in this report have been given opportunities 
to conduct workshops and make contacts at those events. 

Two dozen "collaboration-oriented" workplace educators based 
in New York State. Massachusetts. Canada. New Tersev. and Maine 
have met several times in Albany and Boston in 1993 and 1994. In 
those meetings, they have updated each other on their respective 
projects, established a working set of principles to guide them, and 
identified several ways they might communicate and support each 
other in the future. These activities include establishment of an 
electronic bulletin board for information-sharing and follow-up 
conferences. These activities are seen as a vehicle for helping 
members of this informal network to refine and expand their own 
theory and repertoire of practices, while building a constituency of 
like-minded practitioners who might shape workplace education 
policy (Jurmo, September 1993; Jurmo, April 25, 1994; Jurmo, June 
1994). 

The Skill Builders is a consortium of workplace education 
providers in the Portland, Oregon area. It has developed a 26-week 
training program for adult educators and trainers interested in 
operating workplace basic skills programs. The course mixes 
theoretical and practical classroom activities with a three-month 
practicum. As such, it is one of the few attempts nationally to 
provide training specifically for workplace educators beyond a one- 
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time, three-day workshop (W. Werbel, personal communication, 
April 29, 1994). 



The Adult Literacy Resource Institute in Boston has since 1990 
hosted two informal study groups of workplace educators. The eight 
members of one group meet periodically to discuss ethnographic 
studies of workplaces, in order to better understand workplace needs 
and explore how ethnographic research methods might be adapted 
for use by workplace educators. The second is a "teacher sharing" 
group in which workplace education instructors discuss instruction- 
related theory and practice (D. Rosen, personal communication, April 
29, 1994). 

Canadian researcher Glenda Lewe has prepared a paper for the 
National Center on Adult Literacy for workplace educators interested 
in working in a TQM workplace. She maps out the distinguishing 
characteristics of a TQM organization and identifies the implications 
of each for workplace educators. She argues that, if educators are 
not aware of both the opportunities and constraints of a TQM 
environment, they are likely to design inappropriate curricula and 
otherwise run into trouble (Lewe, in press). 

Lewe advises workplace educators to familiarize themselves 
with TQM, by attending "quality" workshops and joining related 
organizations. She says that educators need to understand that a 
true TQM workplace doesn't do things piecemeal, but introduces a 
number of innovations simultaneously. Educators can also tap into 
educational institutions including many community colleges — 
.which are themselves adopting TQM practices and providing training 
around the issue. 

Lewe notes that educators should look at health care in 
particular for an industry now adopting TQM in a major way. She 
says that TQM assesses success not by ranking individuals but by 
looking at performance of the overall organization. Workplace 
education evaluations might thereby be structured similarly (G. 
Lewe, personal communication, April 8, 1994). 

In Canada, ABC CANADA (based in Toronto) and the Centre for 
Literacy (based in Montreal) have sponsored two four-day summer 
institutes for workplace educators in 1993 and 1994. The 1994 
conference agenda focused in particular on "the collaborative 
approach to evaluation." 
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The ERIC Clearinghouse on Adult. Career, and Vocational 
Education at Ohio State University has recognized the emergence of 
the collaborative perspective described in this report as an 
alternative to the functional context perspective. The Clearinghouse 
has solicited reports and articles which can document the 
development of this perspective. Suitable materials will be included 
in the ERIC database, to enable others to learn from them (S. Imel, 
personal communication, February- 18, 1994). 

Emergent literature in support of collaborative learning 

As described in Chapter I and elsewhere in this report, since 
approximately 1991 there has emerged from workplace literacy 
efforts a different way of interpreting "functional context." In this 
new view, "the workplace context" we should be trying to create is 
one which emphasizes not just technical efficiency and 
competitiveness, but collaborative decision-making, continuous 
learning, and a high quality of work life. "Being functional" in such a 
context means being an active participant in the creation of such a 
new way of organizing work. Those proposing this new 
interpretation of the workplace context argue that workplace 
educators now have to "rethink and restructure." That is, the field 
needs to rethink what it is education programs should be trying to 
accomplish. We then need to restructure education programs to 
reflect the principles and practices of that new workplace. 

While some might assume that this new interpretation of 
contextualized learning has little in common with earlier 
interpretations, this is not the case, as illustrated in the following 
quotations from early advocates of the functional context 
perspective: 

From Thomas Sticht (1991) : 

, Evaluation in the NWLP is not something that is accomplished 
at the end of a program development and implementation 
effort to "see if it worked." Rather, evaluation is an integral 
part of the original design of the program and an ongoing 
process that can permit decisions about how well the program 
is achieving one or more of the purposes of the NWLP and, 
where desirable, to improve the program and its value to adult 
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learners, other partners in the project, and the society at large 
(p. 3). 



. . . it is not true that all aspects of productivity are directly 
mediated by literacy ability. . . if there is simply a blanket 
assumption that increasing literacy ability will increase 
productivity in some unspecified manner, it may not be 
possible to demonstrate that the program has, indeed, 
increased productivity (p. 5). 

. . . program administrators and teachers are concerned with 
meeting the needs of their adult learners and partners. They 
are less concerned, if at all, with meeting the needs of federal 
funding agencies for information for decision making. . . (this) 
problem is compounded by the fact that ... no none is satisfied 
that (existing standardized tests and oher assessment 
instruments) are actually obtaining valid infomation about 
"true" achievements. . . it is (thus) understandable why 
workplace literacy operators, teachers, and adult learners may 
be reluctant to submit to examinations that they feel are 
intrusive and nonrepresentative of what they are teaching and 
learning (p. 8). 

. . . (outside evaluators indicate that what) is needed is 
convincing evidence that useful learning outcomes are being 
achieved . . . and that this new learning results in improved 
productivity in finding, retaining, performing, or advancing in 
a job in the workplace. While various types of ratings (e.g., 
supervisor ratings of increased productivity, teacher ratings of 
improvement, adult learner ratings of pre-and post-program 
increases in learning or productivity) provide useful indicators 
of the program's effects on learning and productivity, such 
ratings are not totally convincing. They are not free of the 
potential for self-deception that may bias ratings. It is the 
desire to overcome these kinds of subjective judgments . . . that 
lead . . to calls for "objective," "empirical," "measurable," 
outcomes of literacy learning and productivity (pp. 8-9). 

The only way to know if growth has taken place is to measure 
the abilities at the outset of the program, and then again later 
on (p. 9). 




(Evaluation) should be undertaken in the spirit of inquiry ~ 
always questioning, seeking information, and using that 
information to modify programs to make them more effective 
Programs that seek to instill the love of lifelong learning in the 
workforce by starting learners off with the first steps into 
workplace literacy, should themselves exhibit positive attitudes 
toward learning ~ learning what they are doing, how they are 
doing it, and what might be done to improve what they are 
doing. Programs that hope to make critical thinkers of others 
should become models of critical thinking themselves. Good 
evaluation requires critical tMnking, continuous learning, and 
thorough documentation to permit others to properly place a 
high value on good works (p. 10). 

From Torie Philippi (1992) : 

America is trying to transition from outdated hierarchical work 
systems to quality-oriented, flattened organizations . . . the 
workplace has become a vast learning community with 
immediate and critical needs (p. 3). 

The most successful . . . programs . . are built on actual job 
scenarios and . . . utilize job materials as a vehicle to teach the 
metacognitive and cognitive strategies that competent workers 
use . . . Instructional objectives are not content-specific, but 
rather focus on the thinking processes that are embedded in 
real-life job task examples with which workers are familiar. . . 
For the individual, (this) ensures that the instruction will be 
meaningful in terms of prior knowledge structures; using 
existing "mental hooks," or schemata, derived from work 
environment and experience to attach new information helps 
ease the process of assimilating old and new knowledge . . .(pp. 
5-6). 

Evaluation of workplace literacy programs should emphasize 
formative evaluation while including cyclical summative 
evaluations (based on the premise that the workplace will 
continue to experience accelerating change and training in new 
skill applications will be an ongoing need) (p. 8). 

Evaluation of workplace literacy programs needs to measure 
program effectiveness independentiy, i.e., by comparing each 
actual program to its own stated goals, rather than comparing it 
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to other programs. This is because each workplace literacy 
program designed in response to company and worker needs is 
unique; no two organizations have exactly the same set of 
critical job tasks to be addressed by instruction (p. 8). 

By the mid-1970s, a new version of independent evaluation 
had developed, called "responsive evaluation" in which the 
evaluator is primarily concerned with program effects in 
relation to the interests and goals of the program 
"stakeholders" . . . Responsive evaluation begins by identifying 
the goals each stakeholder group has for the program. This is 
followed by evaluator observations of program materials and 
implementation, which are compared to the various goal 
expressions. The evaluator identifies issues and concerns, 
collecting data to satisfy these with a variety of instruments. 
Collected information is organized into themes and issues, and 
concerns are matched to audiences. "Progressive focussing" can 
be used to systematically reduce the breadth of inquiry and 
concentrate attention on emerging issues. . . Responsive 
evaluation has potentially strong features for evaluating 
workplace literacy training because it attempts to take into 
account the interests of various groups of stakeholders, rather 
than just the program developers or sponsors, and collect data 
that meets their needs (pp. 8-9). 

. . . programs should be viewed as an integral part of 
organizational training ~ a long-term investment in the 
building of a highly skilled labor force. Evaluation design, as a 
part of such programs, needs to determine program 
effectiveness according to the value yardstickes that business 
and industry apply to training. Quality indicators, measures, 
and performance standards for individual programs need to be 
identified from the workplace community, rather than from 
educational environments (p. 9). 

(Evaluations need to constantly search) for evidence each 
quarter that demonstrates the value of continuing the training 
program and looking for resources with which to expand it. For 
external providers or recipients of demonstration project funds, 
this requires diligent efforts to develop meaningful workplace 
program indicators that demonstrate invaluable impact on the 
organization as well as individual participants so that the 
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program will be institutionalized, budgeted, and incorporated 
into company training on an ongoing basis (p. 13). 



Fr om Larry Mikulecky (1993 and 1994) : 

. . . program providers need to have clear goals for what they 
want to achieve in the limited time that learners are in class. 
They should also seek ways to extend this time beyond the 
classroom (Mikulecky and Lloyd, 1993, p. 93). 

(Programs would ideally) anchor nearly all learning in a 
variety of meaningful contexts which range from workplace to 
reading for personal goals to preparing for further training . . . 
(part of) a life-long learning program (with) dozens of learning 
experiences . . . over several years . . . Most effective programs . 
. . draw upon every available opportunity for literacy, including 
workplace home, further schooling . . . (Mikulecky, 1994, p.5). 

Interviews, analysis of memos and planning documents, and 
early program observations often reveal that significant 
differences about program goals exist among funders, 
supervisors, instructors, materials designers, and learners. 
Evaluation feedback during early program stages often initiates 
necessary clarification among program planners and 
participants. In some cases, goals are expanded, in some goals 
are refined, and in some new vendors are sought (Mikulecky 
and Lloyd, 1993, p. 16). 

. . . goals for workplace literacy programs include improved 
learner literacy abilities, improved literacy practices at work 
and elsewhere, changed learner beliefs about literacy, self, and 
education, and improved learner productivity on the job 
(Mikulecky and Lloyd, 1993, p. 19). 

Lytle . . . suggested that performance measures (tests and 
exercises) miss a good deal of important information about 
adult literacy learning. In addition to gains in literacy skills, 
adults may change in what they believe, in how they behave, 
and in their aspirations. . . Lytle's conceptual framework was 
adapted to the present workplace literacy project to test the 
importance of these aspects of adult learning and to seek ways 
to enhance learning (Mikulecky and Lloyd, 1993, p. 30). 
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It is possible for workplace lieracy programs to affect not only 
the learners' literacy levels and productivity on the job but also 
literacy in their families. Home literacy activities can both 
benefit the employees' children and increase the employees' 
literacy practice time . . . The effect of literacy programs on the 
children and families of workers is often neglected in 
evaluating program effectiveness, however (Mikulecky and 
Iloyd, 1993, p. 34). 

It was not possible ... to obtain data on the actual output of 
the individual employees involved in training . . . (because the) 
companies do not gather output data for units below that of the 
work-team . . . (Mikulecky and Lloyd, 1993, p. 45). 

(Using employer-gathered productivity indicators) is not of 
great use if sample sizes are small and time between 
assessments is not very long (Mikulecky and Lloyd, 1993, p. 
91). 

(In one plant) supervisors had less education . . . and were less 
familiar with the concept of individual employee evaluation. 
Appointments between individual researchers and supervisors 
for the purpose of rating learners' job performance were 
cancelled for several legitimate reasons. As a result, 
supervisors sometimes rated employees without someone to 
remind them to think carefully about each scale. The resulting 
ratings seemed to reflect a desire to complete the task rapidly 
(e.g., many workers received exactly the same rating on each 
scale) (Mikulecky and Lloyd, 1993, p. 92). 

Programs that draw learners from several different jobs may 
not be able to use supervisor ratings to assess impact on 
productivity. Unle3S jobs have several common tasks, it will 
not be possible to construct scales which can be used for all 
learners. If several different scales need to be constructed for 
several different jobs, the small number of learners in each job 
category is likely to preclude any meaningful statistical 
analysis (Mikulecky and Lloyd, 1993, p. 92). 

The above quotations -- as well as other statements by the U.S. 
Department of Education quoted earlier in this chapter ~ indicate 
that those commonly seen as proponents of a functional context 
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perspective share many of the premises used by advocates of the 
emerging "collaborative" approach. These premises include: 

• Stakeholders have a variety of expectations for programs. 
Programs need to find ways for stakeholders to negotiate and 
agree on feasible, meaningful goals, especially given the limited 
time available to programs. 

• While stakeholders might hope for job-related outcomes, 
including such non-job-related uses of literacy as family- 
literacy activities in a workplace curriculum might encourage 
learners to put in more practice time outside the classroom. 
Such practice can generate learner interest in learning and 
other affective and cognitive characteristics required for any 
literacy activity. 

• In their focus on impact on particular job tasks, workplace 
programs often overlook more-important, fundamental 
objectives. These include changing learner beliefs about self 
and literacy, changes in the strategies used to deal with print, 
and changes in the day-to-day ways the learner uses print. 

• For several reasons, it is not easy — and in some cases 
perhaps impossible — to generate evidence of program impact 
on job performance. Reasons include: (1) companies don't keep 
data on individual job performance (records are for work units 
rather than individuals); (2) the length of programs is too short 
to produce significant change; (3) the size of learning groups is 
too small to permit meaningful statistical analysis of data; (4) 
improved job performance might be due to the Hawthorne 
effect of merely being paid attention to rather than to course 
content itself; and (5) supervisors sometimes don't understand 
the value or mechanics of rating individual workers' job 
performance. 

• Rather than being a simplistic exercise to prove to funders 
that a program worked, evaluation should be seen as a means 
for helping all stakeholders understand, and invest themselves 
in a common effort to improve the organization and 
individuals. This "spirit of inquiry" is consistent with good 
adult education and organizational development practice. 
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It thus appears that proponents of functional context and 
collaborative approaches have significant things in common. This 
suggests that, rather than seeing these perspectives as mutually 
exclusive and adversarial, workplace educators might use these 
commonalities as focal points for further productive discussion. 



Elsewhere in the adult literacy field 

Workplace educators might learn from not only the other 
workplace educators described up to this point in this report, but 
from other adult literacy educators operating in non-workplace 
contexts. In particular, those advocating a collaborative approach to 
workplace education might study those who have been developing 
participatory approaches to assessment, evaluation, and general 
program design. Examples of such efforts are outlined below: 

Alternative evaluation and assessment models 

Lytle and Wolfe (1989) map out the approaches to program 
evaluation and learner assessment used by U.S. adult literacy 
programs. They analyze the strengths and limitations of each 
approach and identify the following ten recommendations for 
improving evaluation and assessment: 

1. Program evaluation . . . should be conducted both externally 
and internally. 

2. Program evaluation should be both formative and summative. 

3. Program evaluation and learner assessment should involve 
learners and staff in a participatory process. 

4. Questions for the design of program evaluation should be 
generated from theory, research, evaluation, and program 
practice. 

5. Program evaluation should involve critical reflection on 
program philosophy and goals. 

6. Program evaluation should give prominence to the processes of 
teaching and learning. 

7. Evaluations should be designed to capture a range of learner 
and program outcomes. 

8. Program evaluation and learner assessment require a variety of 
methods for collecting data over time. 

9. To the extent possible, program evaluation and learner 
assessment should be integrated with program functions. 
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10. Program evaluation should be systematic and systemic, 
enabling stakeholders to make comparisons within and across 
programs and contexts (pp. 61-67). 

Lytle and Wolfe also identify three purposes for evaluation: 

The first purpose is accountability, which some studies 
suggest should be a "mutual" accountability in which 
program sponsors and funders work collaboratively with 
program staff and learners to design and implement 
systems for evaluation that are interactive and that 
generate meaningful data appropriate to all levels of the 
system ... 

The second purpose relates to the improvement of 
practice . . . Evaluation and assessment are forms of 
inquiry and staff development ... In the field of adult 
literacy, learners can take a prominent role in discussions 
about program design and implementation and about staff 
development. 

The third purpose for evaluation is the generation of 
new knowledge about adult literacy education. Program- 
based evaluation studies and learner assessment can 
promote dialogue among practitioners within and across 
programs and can be further enhanced by relationships with 
consultants and evaluators from universities. Networks 
formed around issues and practices of evaluation are in a 
position to play a critical role in setting policy ... (p. 68). 

(For more on alternative literacy assessment, also see: Lytle, 1988; 
Lytle et al, 1989; Wolfe, 1988.) 

Fingeret (1993) focuses on learner portfolios, one form of 
participatory assessment touched on by Lytle and Wolfe (1989). She 
defines portfolios "as a process as well as a product," (p. 3) in which 
learner and instructor analyze and document the learner's evolving 
interests, beliefs, and uses of literacy. 

"The content of portfolios is limited only by our creativity and 
imagination" (p. 2). She quotes Valencia's (p. 339) statement that a 
portfolio can include "written responses to reading, reading logs, 
selected daily work, pieces of writing at various stages of completion, 
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classroom tests, checklists . . . The key is to ensure a variety of types 
of indicators of learning so that teachers, . . . students, and 
adrninstrators can build a complete picture of the student's 
development." 

Fingeret expands on this list, as follows: 

Portfolios can include examples of literacy practices outside of 
school, such as copies of drivers licenses, copies of the title 
pages of books that parents read to their children, copies of 
letters written to friends and family, and examples of math, 
reading and writing done in church, civic association meetings, 
or on the job. Portfolios can include copies of journal pages in 
which students discuss their learning (p. 2). 

Fingeret says that "portfolio assessment has four major areas of 
reflection and decision-making, within each there are a number of 
steps; this is a cyclical process, however, rather than a linear one" (p. 
3). The four stages of decision-making in Fingeret's process are 
summarized below: 

STAGE ONE: Decide whether portfolio assessment is consistent 
with vour approach to instructi on and assessmen t. 
The practitioner clarifies his or her beliefs about literacy, the 
teacher-student relationship, assessment, the relationship 
between assessment and instruction, and whether portfolio 
assessment is consistent with those beliefs. 

STAGE TWO: Plan portfolio assessment . 
The practitioner decides what areas to assess, what types of 
materials to collect as evidence, a schedule for developing the 
portfolios, criteria for «\ '.ecting materials to include, the process 
for putting materials into the portfolios, and criteria and a 
process for assessing the contents. 

STAGE THREE: Implement portfolio assessment . 
The practitioner introduces the portfolio concept early on, if 
possible; and then implements the above process of collecting 
and assessing materials. 

STAGE FOUR: Evaluate the process and revise it for future use . 
With learners, the practitioner reflects on the process used and 
they revise it for future use. 

87 

79 



This approach to individual assessment is remarkably similar 
to the collaborative evaluation process for workplace programs 
described above under "Changing funding guidelines" and 
"Evaluation." (See Sperazi and Jurmo, 1994a; and Jurmo and 
Folinsbee, 1994 for details of that process.) It also relates closely to 
the processes being developed for individual needs analysis in 
workplace programs as described under "Analyzing individuals' 
needs, interests, and abilities," above. 

The participatory approach to literacy education 

Since the mid-1980s, there has been a growth in interest 
within the U.S. literacy field in what has come to be known as a 
"participatory" approach to literacy education. In such an approach, 
programs are structured to provide opportunities for learners to play 
active roles in planning and implementing a wide range of activities 
more traditionally carried out by instructors and administrators. 
Learners thereby take on greater degrees of control, responsibility, 
and reward vis-a-vis program activities. Advocates argue that such 
an approach improves efficiency of program operations while 
promoting learners' personal development and their abilities to 
participate in the development of their communities and the larger 
society (Fingeret and Jurmo, 1989). 

As stated in Chapter I, some literacy educators who have used 
a participatory approach in community settings have adapted 
participatory principles and practices for use in workplace education 
settings. A common element in both cases is the notion of shared 
decision-making. In community-based programs, the sharing is done 
primarily by learners and instructors; in workplace settings, the 
number of stakeholders is increased to include management and 
labor representatives, as well. 

Some advocates of a functional context approach to workplace 
education have argued that the participatory approach developed in 
community settings has no place in a workplace. Such an 
interpretation of the participatory approach is presented below: 

. . . notions of learner participation are exaggerated in 
workplace contexts. In a company, they say, it's not just what 
learners want, it's what that company's mission, goals, and 
aspirations are. It's a bottom line issue. Workplace programs 
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must have an organizational perspective in addition to what 
learners need. "You can't apply Paulo Freire's theories of the 
oppressed in business and industry," says Sticht. "You can't say 
'you're oppressing your workers and I'm here to empower 
them to overthrow your dictatorial management styles.' It 
won't work" (Business Council for Effective literacy, January 
1992, p.7). 

Many of those advocating a collaborative alternative to the 
functional context approach would argue that this is a distorted view 
• of the potential of adapting a participatory approach to workplace 
settings. Rather than seeing workers as hostile adversaries and 
discussion of workplace problems as a threat (as has been the case 
in many traditional, "Tayloristic" workplaces), those who have 
adapted participatory principles to workplace settings argue for a 
collaborative relationship between worker-learners and other 
stakeholders. In such a relationship, workers, managers, supervisors, 
union representatives, and instructors work together to analyze and 
improve the workplace. This is seen as an appropriate way of 
helping companies shift toward a team-based, continuous 
improvement management model. 



Within other fields 

In addition to communicating with those developing 
collaborative theory and practice in adult literacy programs 
operating outside the workplace, workplace educators might also 
investigate what participatory-minded professionals are doing in 
disciplines not directly related to adult literacy education. Outlined 
below are two such fields ~ participatory action research and 
training and organizational development. 



Participatory action research 

Innovators in the field of social science research have, since at 
least the 1960s, worked to develop procedures which involve those 
affected by particular social problems in a process of reflection and 
action to solve those problems. These social science procedures do so 
by producing particular kinds of "products" and "processes." The 
"products" in this case are meaningful and reliable information which 
those affected by a problem can use to make the decisions necessary 
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to solve the problem. The "processes" are evaluation and research 
activities which move those involved from being relatively passive 
"providers of information" to more-active "participants" in a 
problem-solving process. 

The roots of this approach to social science research (which 
might be termed "participatory action research") go back to work 
done by anthropologists and sociologists in the 1940s and even 
earlier (L. Sperazi, personal communication, June 22, 1994). William 
Foote Whyte, one of the pioneers of this approach, described the logic 
behind involving informants as active participants, as follows: 

Long before the term became popular, social anthropologists 
and sociologists were carrying out projects which fit under the 
label of participant research. Anyone who serves as a 
participant observer in an organization or community for an 
extended period of time discovers that not all informants 
provide information and ideas of equal value. We always 
encounter one or two individuals who are especially 
knowledgeable, insightful and perceptive regarding the 
dynamics of their organization or community. We do not 
simply give such key informants standard interviews. It is 
useful to the researcher and more enjoyable to the key 
informant if we expand the social process to discuss with these 
individuals what we are trying to find out and also consult 
them about how to interpret what we study. Key informants 
thus become active participants in the research (Whyte, 1991, 
p.9). 

Proponents of this approach suggested that participants not 
only be part of the process of analyzing problems they were facing 
but then also use that analysis to guide actions to solve those 
problems. This focus on using results for action challenged the 
traditional view that use or application of results would compromise 
the scientific legitimacy of what was learned. 

However, proponents argued that social sciences should focus 
on building theories out of careful observation and analysis of human 
behavior rather than with proving already-existing theories. If 
applied to the workplace education field, this concept of "grounded 
theory" (Strauss and Corbin, 1990) would support an evaluation 
strategy which involves stakeholders in a careful analysis of what is 
needed and what practices seem to work best. Such analysis would 
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thereby help workplace educators develop new theory, something 
especially vital in so new a field. 

In the 1960s and 1970s, this approach was applied to 
evaluation of education programs. Kroenbach, for example, argued 
that courses should be improved through evaluation findings 
(Cronbach, 1963). Stake proposed "client-centered" and "responsive" 
evaluation, geared toward helping client groups solve problems 
(Stake, 1967). Guba argued for a "naturalistic" or inductive approach 
to educational evaluation (Guba, 1978). 

Participatory evaluation is commonly used in international and 
community development as a tool for teaching people what they 
need to know to improve their programs and to further build 
community through collective reflection (Kindervatter, 1979; Kinsey, 
1978; Lackey, Peterson, and Pine, 1981). 

"Empowerment" evaluation is a relatively recent variation on 
the participatory research theme. As described by David Fetterman, 
1993 president of the American Evaluation Association and a leading 
proponent of the concept, empowerment evaluation 

... is the use of evaluation concepts and techniques to foster 
self-determination. The focus is on helping people help 
themselves. This evaluation approach is problem focused, 
collaborative, and requires both qualitative and quantitative 
methodologies ... In one form of empowerment evaluation, 
evaluators teach people to conduct their own evaluations and 
thus become more self-sufficient. It desensitizes and 
demystifies evaluations and ideally helps organizations 
internalize evaluation principles and practices, making 
evaluation an integral part of program planning. . . Evaluators 
can serv3 as coaches or facilitators to help others conduct their 
evaluation. They may also conduct the evaluation for the 
group, after the goals and evaluation design have been 
collaboratively established . . . (Fetterman, 1993, p. 115). 

Minneapolis-based non-profit Rainbow Research follows this 
approach in the technical assistance it gives to community 
organizations. In its work with various organizations working on 
housing, domestic violence, and other issues, Rainbow Research has 
developed various "tools" (e.g., an evaluation "Toolbox," a "Program 
Self-Evaluation Tool") which stakeholders use to assess themselves. 
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The organization also produced a report on how to communicate 
evaluation findings "so that they get read and used." Special efforts 
are made to develop evaluation procedures which are appropriate to 
the culture, language, and prior academic experience of stakeholders. 
Evaluation is seen as a way of affirming stakeholders' strengths and 
identifying needed improvements, rather than judging stakeholders' 
deficits. Rainbow Research's "guiding principles" state, among other 
things, that 

Evaluation is most effective when it employs dialog, sharing of 
information, and concern for the long term rather than short 
term . . . and is most effective when (stakeholders) are 
collaborators in the evaluation process rather than objects of 
study (S. Ramirez, personal communication, April 20, 1994). 

Training and organizational development 

A number of organizations have been developed since the mid- 
1970s to provide technical assistance and an "identity" to those 
managers, union representatives, and others trying to transform 
workplaces in "high-performance" directions. These organizations 
have until now generally focused on aspects of organizational change 
other than employee basic skills, but this issue is increasingly being 
put on the agendas of their conferences and other activities. 

Summarized below is information about a sampling of these 
organizations: 

Association for Quality and Participation : AQP began in the late 
1970s as the International Association of Quality Circles. Within 
eight years, members had realized that, to improve American 
workplaces, companies needed to go beyond introducing specific 
innovations like quality circles and instead change their broader 
management philosophy. At that point, the organization changed 
its name to the Association for Quality and Participation. 

The not-for-profit organization provides information and 
technical assistance to human resource directors, quality 
engineers, and others responsible for training employees to 
prepare them for workplace change. AQP promotes a team- 
problem-solving approach to training, in which participants learn 
together and make decisions based on information they generate. 
The goal of such training is to help workplaces transform 
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themselves to contexts in which labor and management work in 
self-directed teams to create knowledge and make decisions. 

AQP sees this work as a "cross between a professional and 
social movement" The association has 10,000 members in 80 
chapters, most of which are in North America. It collects and 
disseminates information via conferences and printed materials. 
(J. Jones, personal communication, April 6, 1994). 

E cology of Work is a yearly series of conferences showcasing 
innovative attempts to re-design workplaces. Human resource 
directors, production managers, and union leaders hear from and 
analyze workplaces attempting to implement "statistical process 
control," "re-engineering," "work-redesign," "TQM" and related 
concepts. One conference is held in Canada, another in the United 
States, the third in Europe, and the fourth ~ for union 
representatives - in various locations (J. Newman, personal 
communication, June 21, 1994). 

Labor studies centers at a number of U.S. universities are doing 
research and providing technical assistance related to workers' 
roles in the changing workplace. In some cases, they are looking 
in particular at the roles of worker basic education and other 
training within these changes. These centers are coordinated by 
the University and College Labor Education Association , whose 
chair is Susan Schurman, director of the Rutgers University Labor 
Education Center (S. Schurman, personal communication, April 12, 
1994). 

The National Workforce Assistance Collaborative is a five-year 
project funded by the U.S. Department of Labor to provide 
technical assistance in the areas of workplace literacy, technical 
training, work-restructuring, and labor-management relations. 
Assistance will be given to "intermediary organizations" serving 
small and medium-sized businesses. In its first year (1993-94), 
the Collaborative is analyzing needs of these businesses and 
intermediary organizations. This analysis will produce case 
studies of ten high-performance organizations, a report on new 
labor-management structures, a guide showing intermediary 
organizations how to serve high performance workplaces, a 
conference on work restructuring, and a report on that conference. 
The project is headed by the National Alliance of Business. Other 
partners include the American Society for Training and 
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Development, the Institute for the Study of Adult Literacy at 
Pennsylvania State University, the Maryland Center for Quality 
and Productivity, and the National Association of Labor 
Management Committees. 
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Chapter 5 



WHAT WE CAN DO 
TO 

DEVELOP APPROPRIATE EVALUATION PRACTICE 



Key stakeholders in the workplace education field have much 
to learn from good work done to date and much work to do to 
develop the field further. Workplace educators, employers, unions, 
f under s and pohcy makers, workers, and others need to study, 
clarify their values, develop action plans, build constituencies, get 
resources, do research and development, break down old barriers, 
and practice the collaborative learning and problem-solving we want 
workplaces to adopt. 



The preceding four chapters indicate that the stakeholders 
involved in workplace education programs have different ideas 
about what those programs should accomplish. The "functional 
.. context" approach became a standard for many programs since the 
later 1980s, and many interpreted this to mean programs should 
focus primarily on reading tasks which learners face on their current 
jobs, as defined by education experts and higher-level managers. 

More recently, workplace educators have developed an 
alternative interpretation of rontextualized learning, one they feel is 
more in keeping with the shifting of workplaces to a team 
management model. In this new, "collaborative" model of workplace 
education, more responsibility for the learning process is placed in • 
the hands of workers, and programs are planned, implemented, and 
monitored by teams composed of learners and other stakeholders. 

Just as the overall models for workplace basic skills education 
have been evolving, so have the approaches used to evaluate them. 
Until recently, programs have tended to rely on some combination of 
informal, internal monitoring and more formal evaluation by an 
"outside expert." Those advocating a collaborative approach to 
programming have placed greater responsibility for evaluation in the 
hands of learners and the stakeholder teams which manage the 
program. In so doing, all stakeholders are to get a greater 
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understanding of and ownership for the program, and be able to 
respond quickly to program needs. 

Chapter 4 describes workplace educators who have taken a 
lead in developing the collaborative approach. They ?se revising 
funding guidelines; developing new assessment, curriculum, and 
evaluation tools; creating new opportunities for staff development; 
and building a constituency for workplace education. Also described 
are others outside the workplace education field per se ~ literacy 
educators, social scientists, and organizational development and 
training specialists •- who are developing tools of possible use to 
workplace literacy specialists. 

Those supporting the creation of new approaches to workplace 
education now have like-minded people and other resources to work 
with and, therefore, reason to be encouraged. However, much mgre 
work must be done to build on the foundation of theory and practice 
which is being put in place. Presented below are some tangible steps 
which key stakeholders might take to support these developments: 

Workplace educators 

"Workplace educators" include both service providers and 
researchers. They might: 

1. Do some reading . On your own, study what others are already 
doing to create new and improved approaches to workplace 
education. Read the sources cited in this report. Recognize the 
"paradigm shift" now going on in the world of organizational 
development and re-think how workplace education might play a 
role in that shift. 

2. Learn from others . Talk with others doing this work to get 
guidance on questions you might have and share your own 
experience. Participate in conferences where these issues are 
discussed. Create new learning opportunities for yourself by 
organizing your own conferences, E-mail, or other means of 
communicating with others with similar interests. 

3. Clarify your values and goals and develop an action plan . 
Based on the above investigations, clarify your own values and 
goals related to workplace education. Rather than see workplace 
education as inherently limited to job-specific goals, recognize that 
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workplace education is one of the few contexts where adult 
learners might have a chance to develop critical thinking and 
other important skills they didn't develop in schools or other 
community institutions. Map out an action plan detailing where 
you want yourself and your institution to go in the coming years. 
Be clear about kinds of services you want to provide and the staff 
development, funding development, and other steps you'll need to 
take. 

4. Build a constituency . Workplace literacy education needs more 
res, .^es. We have few friends in high places. We thus need to 
organize ourselves, demonstrate what we can do, and be prepared 
to exert influence to shape policy and funding to get the resources 
we need. When communicating with others, focus not only on 
strengthening your practice but on building a constituency for 
advocacy purposes. Also see this as an opportunity to exchange 
peer support with others who might be out there on their own 
trying to develop new practice. 

5. Get resources . Collectively, make a case to funders and policy 
makers, to get them to support growth o f our field rather than 
regurgitation of limited kinds of practice. Individually, look for 
new sources of support and make a similar case for good quality 
workplace education in funding proposals, in presentations to 
business and union representatives, and in other forums. 

6. Do research and development Plan and carry out new 
approaches to workplace needs assessment, individual 
assessment, curriculum design, evaluation, and other components 
of a solid program. Document and critically analyze what you are 
doing and disseminate your findings to others, to strengthen your 
own practice and that of others. Don't be afraid to compare what 
you are doing with other models, as well. If we don't honestly 
and critically evaluate ourselves, our ideas will remain untested 
rhetoric, not something others should put much faith in. Use this 
experience to build your expertise, so you have a foundation to 
work from. 

7. Break down old barriers . Be willing to challenge your old 
assumptions, try new ideas, and communicate with others (from 
other fields or other philosophical orientations) with whom you 
don't normally communicate. For example, participate in TQM 
training or investigate how the larger field of evaluation has dealt 
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with many of the issues now facing workplace educators. Also, 
rather than think of evaluation and assessment as separate 
entities or distracting requirements, see them as central parts of 
the overall learning process of which "basic skills classes" are only 
one part. 

8. Practice what we preach . If we are trying to get others to 
invest in ongoing learning, work in teams to communicate and 
make decisions, and create "learning organizations," we should do 
likewise. Re-structure your program or department along 
collaborative lines. Create new networks of others interested in 
similar issues. The above activities are additional ways of doing 
so, as well. 

We should also be willing to take some risks, just as we are 
asking learners, employers, union representatives, and funders to 
do when we ask them to invest in workplace education. We need 
to be willing to "stretch" ourselves, try new ideas, and stand up 
for what we believe in. 

Employers 

"Employers" are management representatives concerned about 
employee basic skills issues. They are not monolithic vis-a-vis the 
employee basic skills. issue. They include production managers 
concerned primarily about job performance company-wide, human 
resource managers who focus more on employee morale and job 
satisfaction across departments, and on supervisors who are 
responsible for both production and human resources issues in their 
particular departments. Each has a different reason for being 
concerned. Here are some things you as management 
representatives can do: 

1. Dc some reading. On your own, study what others are already 
doing to create new and improved approaches to workplace 
education. Recognize that, just as companies are changing, 
workplace education providers are likewise changing how they do 
things. Read the sources cited in this report ~ and others - to 
familiarize yourself with what others have been saying and doing 
about workplace education. Consider how workplace education 
might play a role in the "paradigm shift" now going on in North 
American workplaces. 



95 l'}3 



2. Learn from others . Talk with others -- employers, educators, 
union representatives - doing this work to get guidance on 
questions you might have and to share your own experience. Talk 
with others within your own state or industry as well as 
elsewhere. Participate in conferences where these issues are 
discussed. Create new learning opportunities for yourself by 
working with others (e.g., the local Chamber of Commerce, 
university, adult education program) to organize your own 
meetings or other means of communicating with others with 
similar interests and questions. 

3. Clarify your values and goals . Based on the above 
investigations, clarify your own values and goals related to human 
resources and workplace education. Why are you now thinking 
about this issue? What kind of organization are you trying to 
create? What at this stage might a worker education program 
accomplish for your organization? 

4. Conduct a workplace needs assessment : With the help of an 
adult education provider, conduct an organizational needs analysis 
to clarify how employee basic skills "fits" into your organization's 
larger plans for productivity and quality of work life. 

5. Get resources . With others in your industry, community, or 
state, make a case to funders and policy makers, to get them to 
support growth of our field. Individually, look for new sources of 
support and make a similar case for good quality workplace 
education in funding proposals, in presentations to business and 
union representatives, and in other forums. 

6. Do research and deve lo pment Don't expect everything to be 
totally clear and to work perfectly the first time you get into the 
workplace education issue. Take a systematic "R&D" approach as 
you would with other innovations. Do careful analysis of your 
organization's and individual employees' needs. If a basic skills 
initiative is warranted, plan and implement it carefully. 
Document and analyze what you are doing and disseminate your 
findings to others, to strengthen your own practice and that of 
others. 

7. Break down old barriers . Be willing to challenge your old 
assumptions, try new ideas, and communicate with others (from 
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other fields, other philosophical orientations, or other geographic 
areas) with whom you don't normally communicate. 

8. Practice the values you promote elsewhere in your organization . 
If you are trying to promote critical titiinking, quality, teamwork, 
and "learning organization" principles in your organization, 
consider how you might apply those same principles to planning 
of a workplace basic skills program. The preceding seven 
suggestions present possible ways of doing so. 



Unions 

"Unions" are labor organizations whose mission is to ensure a 
high quality of work life and other rights for workers. Historically, 
union representatives have been leaders in promoting the "quality of 
work life" perspective on workplace education described in Chapter 
I. 

1. Do some reading . On your own, study what others are already 
doing to create new and improved aproaches to workplace 
education. Read the sources cited in this report ~ and others -- to 
familiarize yourself with what others in unions and other 
institutions have been saying and doing about workplace 
education. Consider how workplace education might play a role in 
enabling workers to participate actively in shaping the "paradigm 
shift" now going on in North American workplaces. 

2. Learn from others . Talk with others - union representatives, 
educators, and employers ~ doing this work to get guidance on 
questions you might have and to share your own experience. Talk 
with others within your own state or industry as well as 
elsewhere. Participate in conferences where these issues are 
discussed. Create new learning opportunities for yourself by 
working with others (e.g., the s u ate AFL-CIO, university labor- 
studies departments, adult education community) to organize your 
own meetings or other means of communicating with others with 
similar interests and questions. 

3. Clarify your values and goals . Based on the above 
investigations, clarify your own values and goals related to 
workplace education. Why are you thinking about this issue? 
What are you trying to accomplish for the workers you serve? 
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What roles might a worker education initiative play in your larger 
efforts? 



4. Conduct a workplace needs assessment : With the cooperation of 
your company's management and with the help of an adult 
education provider, conduct an organizational needs analysis to 
clarify how employee basic skills "fits" into the company's larger 
plans for productivity and quality of work life. 

5. Get resources . With others in your industry, labor community, 
local community, or state, make a case to funders and policy 
makers, to get them to support growth of our field. Individually, 
look for new sources of support and make a similar case for good 
quality workplace education in funding proposals, in presentations 
to union and business representatives, and in other forums. 

6. Do research and development . Don't expect everything to be 
totally clear and to work perfectly the first time you get into the 
workplace education issue. Take a systematic "R&D" approach as 
you would with other innovations. Do careful analysis of the 
needs of your workplace and individual members. If a basic 
skills initiative is warranted, plan and implement it carefully. 
Document and analyze what you are doing and disseminate your 
findings to others, to strengthen your own practice and that of 
others. 

7. Break down old barriers . Be willing to challenge your old 
assumptions, try new ideas, and communicate with others (from 
other fields, other philosophical orientations, or other geographic 
areas) with whom you don't normally communicate. 

8. Practice the union values vou promote elsewhere . If your 
union promotes the value of organizing and collective action as 
means to effecting change, consider working collaboratively to 
create a solid constituency and resource base for high-quality 
worker education. The preceding seven suggestions present 
possible ways of doing so. 



Funders and policy makers 

"Funders and policy makers" in this case include local, state, 
and federal agencies giving funds and shaping guidelines for 
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workplace education. This category also includes private-sector 
funders (for example, foundations, labor organisations, trade 
associations) of workplace education programs and policy institutes 
which shape policy for the field. For some funders and policy 
makers, the following recommendations will be "on target." Others 
will already have carried out some or all of the recommendations. 

1. Do some reading . On your own, study what others are already 
doing to create new and improved aproaches to workplace 
education. Read the sources cited in this report - and others - to 
familiarize yourself with what others have been saying and doing 
about workplace education. Go beyond the sources and thinking 
commonly cited by funders and policy makers and study some of 
the new thinking and forms of practice which have emerged from 
research and development in the 1990s. 

2. Learn from others . Talk with others — employers, educators, 
union representatives — doing this work to get updated views not 
necessarily captured in the literature. Participate in conferences 
where these issues are discussed, but not necessarily in the "top- 
down" role commonly expected of funders and policy makers. 
That is, rather than allowing yourself to be placed on a podium in 
the role of source of knowledge and money, create situations 
where there can be more of a genuine dialogue. In such forums, 
ask for ideas and feedback from the field and be available to 
respond. Do this in the same spirit of continuous improvement 
and continuous learning which we are now asking others to adopt. 

3. Clarify your values and goals . Based on the above 
investigations, update your own values and goals related to 
workplace education. Has your thinking on this issue changed? If 
so, why? Are you now considering some of the arguments and 
experience gained in the field in recent years, or are you stuck 
with the same thinking which policy makers put in place when 
workplace education first got rolling in the mid-1980s? Should 
the field be shoring up old workplaces or creating new ones? 

Re-consider what your role in the field should be. Are you a 
"dispenser of gifts" or an active partner in the field? Do you tell 
others what to do or listen to what the field says it needs and 
work with the field to shape policy through collaborative decision- 
making? 
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4. Re-think what programs might accomplish and what they need 
to do so . Move away from focusing workplace education so 
heavily on limited tasks like "reading of manuals." Consider other 
focal points like "collaborative decision-making," "generating 
interest in lifelong learning," and "oral communications in a 
multicultural environment" 

Reconsider how program impact is measured. (Don't think only 
in terms of "numbers" and "changing individuals." Look for more- 
authentic evidence of impact on learners. And see whether the 
organization is changing, too.) 

See "curriculum" not as a set of "materials" to be taught to 
everyone in a "class." See it instead as a process in which 
learners, instructors, and other stakeholders continually set goals, 
carry out learning activities, and monitor progress. A curriculum 
might have many different types of learning activities, some very 
individualized. See "curriculum" — an ongoing process of setting 
learning goals and finding ways to achieve them — as the thread 
which holds the "learning organization" together. 

5. Revise guidelines and get more resources . With other 
stakeholders from business, labor, education, and other sectors, 
revise your funding guidelines for workplace education programs 
to reflect feedback from the field. Don't allow funding to be used 
for quick fixes which don't produce lasting benefit for individual 
workplaces or the field as a whole. Recognize that investment in 
pilot programs is now paying off, and continue to find public- and 
private-sector resources to support effective research and 
development. 

Rather than feeling compelled to fund large-scale, longer-term 
projects, funders might consider issuing smaller grants to local 
partnerships to enable them to first carry out careful workplace 
needs aseessments. Based on the results of those assessments, 
funders can then decide whether a basic skills program is truly 
warranted. 

Promote private-sector investment in pilot projects (like the 
special projects carried out by the banking, printing) and home- 
building industries) in funding guidelines, in presentations to 
business and union representatives, and in other forums. 

6. Do research and development . Recognize that research and 
development done to date is producing good evidence which 
others can now learn from. If you are funding demonstration 
projects, encourage some real experimentation rather than 
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requiring ail programs to fit one particular mold. Encourage 
creative solutions to the problems the field has now identified. Be 
sure that those results get disseminated in effective ways rather 
than being filed away. (Consider funding a new organization like 
the Business Council for Effective Literacy, which focused on 
prompt, customized cttssemination of information.) Create 
opportunities (e.g., workshops, study groups, networks, E-mail, 
newsletters) for practitioners to study others' experience and 
document and learn from their own. 

7. Break down old barriers . Be willing to challenge your old 
assumptions, try new ideas, and communicate with others (from 
other fields, other philosophical orientations, or other geographic 
areas) with whom you don't normally communicate. Open up 
communication within the field by funding conferences, 
newsletters, and electronic communication. Look beyond "turf" 
and the personalities of those involved in the field and think more 
deeply and creatively. 

8. Practice the values vou promote elsewhere . If you are now 
encouraging the workplace education field to think in terms of 
"quality," "long-term investment in education," "teamwork," and 
"critical thinking," then you might consider how you might apply 
those same principles to your own operations and to your 
relationship with others in the field. The preceding seven 
suggestions present possible ways of doing so. 

Workers 

Workers aren't normally thought of as active participants in 
the workplace education field. Instead, they are too often seen as 
"recipients" of services provided by others 

Those normally considered providers of those services — 
educators, employers, and unions ~ might now find new ways of 
encouraging workers (especially those who have now participated in 
workplace education programs) to become more involved in 
developing the field. Workers would do so not only for themselves 
but for the many other fellow workers who might benefit. 

Here are some ways in which workers might get involved: 
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1. Enable learners to communicate with other learners . In some 
workplace education programs, there are few opportunities for 
learners to communicate with other learners in the same class or 
in other classes being provided. In such a case, "student councils" 
or "learner forums" might be established to enable learners to 
discuss education-related issues. They might, for example, clarify 
what they hope programs will accomplish, what they like about 
programs and what they don't, what they and others can do to 
support worker education, and what learners can do to pursue 
further educational and job opportunities. Similar forums might 
be set up to enable learners from various worksites or from local 
community-based programs to communicate, as well. Such 
communication channels have already been established by 
community-based literacy programs, to enable learners to provide 
peer support and shape program policy. If community-based 
programs can do it, why not workplace programs? 

2. Enable learners to communicate with other workers . 
Participants in workplace education programs might be given 
opportunities to speak about education issues with other workers 
not currently involved in any education program. Learners can 
explain the benefits of participation and suggest ways in which 
other workers might get involved, either as participants and/or as 
supporters of the program. 

3. Involve learners in policy development . Participating workers 
can play valuable roles in shaping workplace education policies of 
their employers, their unions, and outside funding agencies. They 
can do so by participating in workplace education meetings and 
conferences, public hearings, and news coverage; writing letters; 
and inviting internal and external policy makers to visit classes. 

4. Involve learners in staff development . Learners can serve as 
resource persons in staff development activities. They can help 
set criteria for selection of education staff, participate in staff 
selection and training activities, and give feedback about staff 
performance. 

5. Involve learners in design and implementation of curricula . 
Learners can be encouraged to play active roles in all stages of 
designing and implementing instructional activities. They can 
help clarify learning objectives, serve as peer-instructors, and 
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otherwise be seen as active participants in learning rather than 
passive recipients. 



6. Involve learners in program evaluation . Learners can give 
valuable input into program evaluations, especially when those 
evaluations are carried out by site teams composed of a range of 
program stakeholders. 



In conclusion 

The workplace education field now has had more than ten 
years of experience to learn from. We are identifying new needs and 
opportunities for workplace learning and developing new program 
models and practical tools. In the process, we are questioning old 
assumptions, refining some of them and setting some aside 
altogether. 

There is a danger that, rather than seeing these developments 
as opportunities for creating a new and improved field, we will set 
up camps organized around the thinking we feel most comfortable 
with. If that happens, we will fail to make use of the learning 
opportunities which the last ten years have provided us with. 

Instead, we need to foster an ongoing dialogue in which we can 
critically examine what we are doing, learn from it, and create new 
ways of blinking and practice. Those who might feel a bit worn out 
from the flurry of the past ten years should also encourage 
newcomers to be part of that dialogue. 

Jorie Philippi acknowledged the need to look for new solutions 
as follows: 

Workplace literacy, although quickly burgeoning, is still a 
new field. There are still no "givens" in an organization's 
literacy problems, no pat solutions, no panaceas, and no quick 
fixes for sale in boxes off the shelf (although some opportunists 
would have us believe there are). Each practitioner must be 
willing to commit to the difficult job of seeing the problem, 
carving out new solutions based on sound learning theory, 
collecting proof of effectiveness for each organization, and 
disseminating the results. Only with hard data gathered from a 
variety of program models, techniques, and users can we begin 



to draw universal conclusions and set national policy about 
best practices (i.e., processes) to apply when solving workplace 
literacy problems. 

The workplace is not a static environment; the rapidly 
accelerating changes that affect it also affect the problems and 
solutions that arise concerning workplace literacy. We need 
practitioners who aren't afraid to seek out ever-changing 
messes, name them, define ihem, research them, and clean 
them up (Business Council for Effective Literacy, July 1993, p. 
10). 
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APPENDIX: 



Key organizations and individuals 
cited in this report 

ORGANIZATIONS 

ABC CANADA 

1450 Don Mills Rd. 

Don Mills, Ontario M3B 2X7 

416/442-2292 

Contact: Sue Folinsbee 

Adult Literacy Resource Institute 

989 Commonwealth Ave. 

Boston, MA 02215 

617/782-8956 

Contact: David Rosen 

Applied Behavioral and Cognitive Sciences, Inc. 
2062 Valley View Rd. 
a Cajon, CA 92019 
Contact: Thomas Sticht 

Association for Quality and Participation 
801-B West 8th St. 
Cincinnati, OH 45203 
513/381-1959 

Contacts: Joanne Jones and Sara Olberding 

Center for Working Life 
3814 SE Martins St. 
Portland, OR 97202-7645 
503/774-6088 
Contact: Lee Shore 

Center on Education and Training for Employment 
(See ERIC Clearinghouse.) 
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City University of New York 

Office of Academic Affairs 

555 W. 57th St. 

Room 1512 

New York, NY 10019 

212/794-5556 

Contact: Ann Heininger 

Civil Service Employees Association 
1 Lear Jet Lane 
Ste. 3 

Latham, NY 12110-2393 
518/785-4669 
Contact: Ira Baumgarten 

Connecticut Business and Industry Association 
(See Evaluation Research, below.) 

Ecology of Work 

c/o Conference Support Systems Unlimited 
Box 44A 

Northwood, NH 03261 
603/942-8189 

Contacts: Tom Chase and Jan Newman 

El Paso Community College 
The literacy Center 
PO Box 20500 
El Paso, TX 79998 
915/534-4617 

Contacts: Ann Savino, Roland Chanove, and Kay Taggart 

ERIC Clearinghouse on Adult, Career, and Vocational 

Education 
1900 Kenny Rd. 
Columbus, OH 43210-1090 
800/848-4815 

Contacts: Susan Imel, Sandra Kerka, and Sandra Pritz 
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Evaluation Research 
130 Warren St. 
Newton Center, MA 02159 
617/527-6081 
Contact: Laura Sperazi 

Literacy Partnerships 
14 Griffin St. 

East Brunswick, NJ 08816-4806 

908/254-2237 

Contact: Paul Jurmo 

Massachusetts Workplace Education Initiative 

Mass. Dept. of Education 

Adult & Community Learning Services 

350 Main St. 

Maiden, MA 02148 

617/388-3300, Ext. 430 

Contact Mr. Johan Uvin 

National Workforce Assistance Collaborative 

c/o Institute for the Study of Adult Literacy 

Pennsylvania State University 

204 Calder Way, Suite 209 

University Park, PA 16801-4756 

814/863-3777 

Contact: Eunice Askov 

National Workplace Literacy Program 

U.S. Department of Education 

Office of Vocational and Adult Education 

Division of Adult Education and Literacy 

Washingto, DC 20202-7240 

Contact: Sarah Newcomb 

New York State Education Department 

Education Bldg., Room 315 

Albany, NY 12234 

518/474-5506 

Contact: Robert Knower 



A.3. 

115 



New York State Governor's Office of Employee Relations 

Corning Tower, 23 rd Floor 

Albany, NY 12223 

518/474-6704 

Contact: Diane Wagner 

Rainbow Research 
621 West Lake St. 
Minneapolis, MM 55408 
612/824-0724 
Contact: Steven Mayer 

REACH 

(See New York State Governor's Office for Employee Relations and 
Civil Service Employees Association.) 

The Skill Builders 
Portland Community College 
PO Box 19000 
Portland, OR 97280 
503/244-6111, ext. 6287 
Contact: Wayne Werbel 

Texas Workforce Education Consortium 
(See El Paso Community College.) 

University and College Labor Education Association 

c/o Labor Education Center 

Rutgers University 

Ryders Lane and Clifton Ave. 

New Brunswick, NJ 08903 

908/932-1748 

Contact: Susan Schurman 

University of Tennessee 
Center for Literacy Studies 
2046 Terrace Ave. 
Knoxville, TN 37996-3400 
615/974-4109 
Contact: Juliet Merifield 
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Workforce Instructional Network 

Southwest Texas State University 

601 University Dr. 

San Marcos, TX 78666-4616 

512/245-8142 

Contacts: Jonathan Engel and Pamela McBride 



INDIVIDUALS 

Judith Alamprese 

COSMOS Corporation 
1735 Eye St., NW 
Ste. 613 

Washington, DC 20006 
202/728-3939 

Kathryn Chang Barker 

Canadian Labour Force Development Board 

23-66 Slater St. 

Ottawa, ON KIP 5H1 

Canada 

613/230-6264 

Hanna Fingeret 
Literacy South 
Snow Building, Rm. 202 
331 W. Main St. 
Durham, NC 27701 
919/682-8108 

Sue Folinsbee (Waugh) 

(See ABC CANADA, above.) 

Sheryl Gowen 
Asst., Director 

Center for the Study of Adult Literacy 
Georgia State University 
University Plaza 
Atlanta, GA 30303-3083 
404/651-2405 

A.5. 
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Glynda Hull 

National Center for Research in Vocational Education 
University of California at Berkeley 
1995 University Ave., Ste. 375 
Berkeley, CA 94704 

Susan Imel 

(See ERIC Clearinghouse, above.) 
Sandra Kerka 

(See ERIC Clearinghouse, above.) 

Glenda Lewe 
Glenda Lewe Consulting 
530 Laurier Ave. West 
Suite 2301 
Ottawa, ON KIR 7T1 
Canada 

613/233-3783 
Susan Lytle 

National Center on Adult Literacy 
University of Pennsylvania 
3910 Chestnut St. 
Philadelphia, PA 19104-3111 
215/898-2100 

Muriel Medina 
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